System and method for performing motion capture and image reconstruction

ABSTRACT

A system and method are described for performing motion capture on a subject. For example, a computer-implemented method according to one embodiment of the invention comprise: creating a scalar field for the three-dimensional (3-D) capture volume of the subject; generating a surface mesh for the scalar field; retaining good vertices and removing bad vertices of the surface mesh; and storing the good vertices for use in subsequent reconstruction of the motion of the subject. Another computer-implemented method comprises: capturing a series of image frames of the subject over a period of time each frame each frame having a plurality of vertices defining a captured surface of the subject; establishing a reference frame having one or more of the plurality of vertices; performing frame-to-frame tracking to identify vertices within the N′th frame based on the (N−1)′th frame or an earlier frame; and performing reference-to-frame tracking to identify vertices within the N′th frame based on the reference frame to counter potential drift between the frames. Yet another computer-implemented method comprises: capturing motion capture data including a plurality of images of the N vertices during a motion capture session; retrospectively identifying X of the N vertices to track across the plurality of images where X&lt;N; and tracking the X vertices across the plurality of images.

PRIORITY CLAIM

This application claims the benefit of U.S. Provisional Application No.60/834,771 entitled, “System and Method F or Performing Motion”, filedon Jul. 31, 2006.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to the field of motion capture. Moreparticularly, the invention relates to an improved apparatus and methodfor performing motion capture and image reconstruction.

2. Description of the Related Art

“Motion capture” refers generally to the tracking and recording of humanand animal motion. Motion capture systems are used for a variety ofapplications including, for example, video games and computer-generatedmovies. In a typical motion capture session, the motion of a “performer”is captured and translated to a computer-generated character.

As illustrated in FIG. 1 in a motion capture system, a plurality ofmotion tracking “markers” (e.g., markers 101, 102) are attached atvarious points on a performer's 100's body. The points are selectedbased on the known limitations of the human skeleton. Different types ofmotion capture markers are used for different motion capture systems.For example, in a “magnetic” motion capture system, the motion markersattached to the performer are active coils which generate measurabledisruptions x, y, z and yaw, pitch, roll in a magnetic field.

By contrast, in an optical motion capture system, such as thatillustrated in FIG. 1, the markers 101, 102 are passive spherescomprised of retro-reflective material, i.e., a material which reflectslight back in the direction from which it came, ideally over a widerange of angles of incidence. A plurality of cameras 120, 121, 122, eachwith a ring of LEDs 130, 131, 132 around its lens, are positioned tocapture the LED light reflected back from the retro-reflective markers101, 102 and other markers on the performer. Ideally, theretro-reflected LED light is much brighter than any other light sourcein the room. Typically, a thresholding function is applied by thecameras 120, 121, 122 to reject all light below a specified level ofbrightness which, ideally, isolates the light reflected off of thereflective markers from any other light in the room and the cameras 120,121, 122 only capture the light from the markers 101, 102 and othermarkers on the performer.

A motion tracking unit 150 coupled to the cameras is programmed with therelative position of each of the markers 101, 102 and/or the knownlimitations of the performer's body. Using this information and thevisual data provided from the cameras 120-122, the motion tracking unit150 generates artificial motion data representing the movement of theperformer during the motion capture session.

A graphics processing unit 152 renders an animated representation of theperformer on a computer display 160 (or similar display device) usingthe motion data. For example, the graphics processing unit 152 may applythe captured motion of the performer to different animated charactersand/or to include the animated characters in differentcomputer-generated scenes. In one implementation, the motion trackingunit 150 and the graphics processing unit 152 are programmable cardscoupled to the bus of a computer (e.g., such as the PCI and AGP busesfound in many personal computers). One well known company which producesmotion capture systems is Motion Analysis Corporation (see, e.g.,www.motionanalysis.com).

SUMMARY

A system and method are described for performing motion capture on asubject using fluorescent lamps. For example, a system according to oneembodiment of the invention comprises: a synchronization signalgenerator to generate one or more synchronization signals; one or morefluorescent lamps configured to strobe on and off responsive to a firstone of the one or more synchronization signals, the fluorescent lampscharging phosphorescent makeup, paint or dye applied to a subject for amotion capture session; and a plurality of cameras having shuttersstrobed synchronously with the strobing of the light source to captureimages of the phosphorescent paint, wherein the shutters are open whenthe light source is off and the shutters are closed when the lightsource is on.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained from thefollowing detailed description in conjunction with the drawings, inwhich:

FIG. 1 illustrates a prior art motion tracking system for tracking themotion of a performer using retro-reflective markers and cameras.

FIG. 2 a illustrates one embodiment of the invention during a timeinterval when the light panels are lit.

FIG. 2 b illustrates one embodiment of the invention during a timeinterval when the light panels are dark.

FIG. 3 is a timing diagram illustrating the synchronization between thelight panels and the shutters according to one embodiment of theinvention.

FIG. 4 is images of heavily-applied phosphorescent makeup on a modelduring lit and dark time intervals, as well as the resultingreconstructed 3D surface and textured 3D surface.

FIG. 5 is images of phosphorescent makeup mixed with base makeup on amodel both during lit and dark time intervals, as well as the resultingreconstructed 3D surface and textured 3D surface.

FIG. 6 is images of phosphorescent makeup applied to cloth during litand dark time intervals, as well as the resulting reconstructed 3Dsurface and textured 3D surface.

FIG. 7 a illustrates a prior art stop-motion animation stage.

FIG. 7 b illustrates one embodiment of the invention where stop-motioncharacters and the set are captured together.

FIG. 7 c illustrates one embodiment of the invention where thestop-motion set is captured separately from the characters.

FIG. 7 d illustrates one embodiment of the invention where a stop-motioncharacter is captured separately from the set and other characters.

FIG. 7 e illustrates one embodiment of the invention where a stop-motioncharacter is captured separately from the set and other characters.

FIG. 8 is a chart showing the excitation and emission spectra of ZnS:Cuphosphor as well as the emission spectra of certain fluorescent and LEDlight sources.

FIG. 9 is an illustration of a prior art fluorescent lamp.

FIG. 10 is a circuit diagram of a prior art fluorescent lamp ballast aswell as one embodiment of a synchronization control circuit to modifythe ballast for the purposes of the present invention.

FIG. 11 is oscilloscope traces showing the light output of a fluorescentlamp driven by a fluorescent lamp ballast modified by thesynchronization control circuit of FIG. 9.

FIG. 12 is oscilloscope traces showing the decay curve of the lightoutput of a fluorescent lamp driven by a fluorescent lamp ballastmodified by the synchronization control circuit of FIG. 9.

FIG. 13 is a illustration of the afterglow of a fluorescent lampfilament and the use of gaffer's tape to cover the filament.

FIG. 14 is a timing diagram illustrating the synchronization between thelight panels and the shutters according to one embodiment of theinvention.

FIG. 15 is a timing diagram illustrating the synchronization between thelight panels and the shutters according to one embodiment of theinvention.

FIG. 16 is a timing diagram illustrating the synchronization between thelight panels and the shutters according to one embodiment of theinvention.

FIG. 17 is a timing diagram illustrating the synchronization between thelight panels and the shutters according to one embodiment of theinvention.

FIG. 18 is a timing diagram illustrating the synchronization between thelight panels and the shutters according to one embodiment of theinvention.

FIG. 19 illustrates one embodiment: of the camera, light panel, andsynchronization subsystems of the invention during a time interval whenthe light panels are lit.

FIG. 20 is a timing diagram illustrating the synchronization between thelight panels and the shutters according to one embodiment of theinvention.

FIG. 21 is a timing diagram illustrating the synchronization between thelight panels and the shutters according to one embodiment of theinvention.

FIG. 22 illustrates one embodiment of the invention where color is usedto indicate phosphor brightness.

FIG. 23 illustrates weighting as a function of distance from surface.

FIG. 24 illustrates weighting as a function of surface normal.

FIG. 25 illustrates scalar field as a function of distance from surface

FIG. 26 illustrates one embodiment of a process for constructing a 3-Dsurface from multiple range data sets.

FIG. 27 illustrates one embodiment of a method for vertex tracking formultiple frames.

FIG. 28 illustrates one embodiment of a method for vertex tracking of asingle frame.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Described below is an improved apparatus and method for performingmotion capture using shutter synchronization and/or phosphorescentmakeup, paint or dye. In the following description, for the purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of the present invention. It will be apparent,however, to one skilled in the art that the present invention may bepracticed without some of these specific details. In other instances,well-known structures and devices are shown in block diagram form toavoid obscuring the underlying principles of the invention.

The assignee of the present application previously developed a systemfor performing color-coded motion capture and a system for performingmotion capture using a series of reflective curves painted on aperformer's face. These systems are described in the co-pendingapplications entitled “APPARATUS AND METHOD FOR CAPTURING THE MOTIONAND/OR EXPRESSION OF A PERFORMER,” Ser. No. 10/942,609, and Ser. No.10/942,413, filed Sep. 15, 2004. These applications are assigned to theassignee of the present application and are incorporated herein byreference.

The assignee of the present application also previously developed asystem for performing motion capture of random patterns applied tosurfaces. This system is described in the co-pending applicationsentitled “APPARATUS AND METHOD FOR PERFORMING MOTION CAPTURE USING ARANDOM PATTERN ON CAPTURE SURFACES,” Ser. No. 11/255,854, Filed Oct. 20,2005. This application is assigned to the assignee of the presentapplication and is incorporated herein by reference.

The assignee of the present application also previously developed asystem for performing motion capture using shutter synchronization andphosphorescent paint. This system is described in the co-pendingapplication entitled “APPARATUS AND METHOD FOR PERFORMING MOTION CAPTUREUSING SHUTTER SYNCHRONIZATION,” Ser. No. 11/077,628, Filed Mar. 10, 2005(hereinafter “Shutter Synchronization” application). Briefly, in theShutter Synchronization application, the efficiency of the motioncapture system is improved by using phosphorescent paint or makeup andby precisely controlling synchronization between the motion capturecameras' shutters and the illumination of the painted curves. Thisapplication is assigned to the assignee of the present application andis incorporated herein by reference.

System Overview

As described in these co-pending applications, by analyzing curves orrandom patterns applied as makeup on a performer's face rather thandiscrete marked points or markers on a performer's face, the motioncapture system is able to generate significantly more surface data thantraditional marked point or marker-based tracking systems. The randompatterns or curves are painted on the face of the performer usingretro-reflective, non-toxic paint or theatrical makeup. In oneembodiment of the invention, non-toxic phosphorescent makeup is used tocreate the random patterns or curves. By utilizing phosphorescent paintor makeup combined with synchronized lights and camera shutters, themotion capture system is able to better separate the patterns applied tothe performer's face from the normally-illuminated image of the face orother artifacts of normal illumination such as highlights and shadows.

FIGS. 2 a and 2 b illustrate an exemplary motion capture systemdescribed in the co-pending applications in which a random pattern ofphosphorescent makeup is applied to a performer's face and motioncapture is system is operated in a light-sealed space. When thesynchronized light panels 208-209 are on as illustrated FIG. 2 a, theperformers' face looks as it does in image 202 (i.e. the phosphorescentmakeup is only slightly visible). When the synchronized light panels208-209 (e.g. LED arrays) are off as illustrated in FIG. 2 b, theperformers' face looks as it does in image 203 (i.e. only the glow ofthe phosphorescent makeup is visible).

Grayscale dark cameras 204-205 are synchronized to the light panels208-209 using the synchronization signal generator PCI Card 224 (anexemplary PCI card is a PCI-6601 manufactured by National Instruments ofAustin, Tex.) coupled to the PCI bus of synchronization signal generatorPC 220 that is coupled to the data processing system 210 and so that allof the systems are synchronized together. Light Panel Sync signal 222provides a TTL-level signal to the light panels 208-209 such that whenthe signal 222 is high (i.e. ≧2.0V), the light panels 208-209 turn on,and when the signal 222 is low (i.e. ≦0.8V), the light panels turn off.Dark Cam Sync signal 221 provides a TTL-level signal to the grayscaledark cameras 204-205 such that when signal 221 is low the camera 204-205shutters open and each camera 204-205 captures an image, and when signal221 is high the shutters close and the cameras transfer the capturedimages to camera controller PCs 205. The synchronization timing(explained in detail below) is such that the (camera 204-205 shuttersopen to capture a frame when the light panels 208-209 are off (the“dark” interval). As a result, grayscale dark cameras 204-205 captureimages of only the output of the phosphorescent makeup. Similarly, LitCam Sync 223 provides TTL-level signal to color lit cameras 214-215 suchthat when signal 221 is low the camera 204-205 shutters open and eachcamera 204-205 captures an image, and when signal 221 is high theshutters close and the cameras transfer the captured images to cameracontroller computers 225. Color lit cameras 214-215 are synchronized (asexplained in detail below) such that their shutters open to capture aframe when the light panels 208-209 are on (the “lit” interval). As aresult, color lit cameras 214-215 capture images of the performers' faceilluminated by the light panels.

As used herein, grayscale cameras 204-205 may be referenced as “darkcameras” or “dark cams” because their shutters normally only when thelight panels 208-209 are dark. Similarly, color cameras 214-215 may bereferenced as “lit cameras” or “lit cams” because normally theirshutters are only open when the light panels 208-209 are lit. Whilegrayscale and color cameras are used specifically for each lightingphase in one embodiment, either grayscale or color cameras can be usedfor either light phase in other embodiments.

In one embodiment, light panels 208-209 are flashed rapidly at 90flashes per second (as driven by a 90 Hz square wave from Light PanelSync signal 222), with the cameras 204-205 and 214-205 synchronized tothem as previously described. At 90 flashes per second, the light panels208-209 are flashing at a rate faster than can be perceived by the vastmajority of humans, and as a result, the performer (as well as anyobservers of the motion capture session) perceive the room as beingsteadily illuminated and are unaware of the flashing, and the performeris able to proceed with the performance without distraction from theflashing light panels 208-209.

As described in detail in the co-pending applications, the imagescaptured by cameras 204-205 and 214-215 are recorded by cameracontrollers 225 (coordinated by a centralized motion capture controller206) and the images and images sequences so recorded are processed bydata processing system 210. The images from the various grayscale darkcameras are processed so as to determine the geometry of the 3D surfaceof the face 207. Further processing by data processing system 210 can beused to map the color lit images captured onto the geometry of thesurface of the face 207. Yet further processing by the data processingsystem 210 can be used to track surface points on the face fromframe-to-frame.

In one embodiment, each of the camera controllers 225 and central motioncapture controller 206 is implemented using a separate computer system.Alternatively, the camera controllers and motion capture controller maybe implemented as software executed on a single computer system or asany combination of hardware and software. In one embodiment, the cameracontroller computers 225 are rack-mounted computers, each using a 945GTSpeedster-A4R motherboard from MSI Computer Japan Co., Ltd. (C&K Bldg.6F 1-17-6, Higashikanda, Chiyoda-ku, Tokyo 101-0031 Japan) with 2 Gbytesof random access memory (RAM) and a 2.16 GHz Intel Core Duo centralprocessing unit from Intel Corporation, and a 300 GByte SATA hard diskfrom Western Digital, Lake Forest Calif. The cameras 204-205 and 214-215interface to the camera controller computers 225 via IEEE 1394 cables.

In another embodiment the central motion capture controller 206 alsoserves as the synchronization signal generator PC 220. In yet anotherembodiment the synchronization signal generator PCI card 224 is replacedby using the parallel port output of the synchronization signalgenerator PC 220. In such an embodiment, each of the TTL-level outputsof the parallel port are controlled by an application running onsynchronization signal generator PC 220, switching each TTL-level outputto a high state or a low state in accordance with the desired signaltiming. For example, bit 0 of the PC 220 parallel port is used to drivesynchronization signal 221, bit 1 is used to drive signal 222, and bit 2is used to drive signal 224. However, the underlying principles of theinvention are not limited to any particular mechanism for generating thesynchronization signals.

The synchronization between the light sources and the cameras employedin one embodiment of the invention is illustrated in FIG. 3. In thisembodiment, the Light Panel and Dark Cam Sync signals 221 and 222 are inphase with each other, while the Lit Cam Sync Signal 223 is the inverseof signals 221/222. In one embodiment, the synchronization signals cyclebetween 0 to 5 Volts. In response to the synchronization signal 221 and223, the shutters of the cameras 204-205 and 214-215, respectively, areperiodically opened and closed as shown in FIG. 3. In response to syncsignal 222, the light panels are periodically turned off and on,respectively as shown in FIG. 3. For example, on the falling edge 314 ofsync signal 223 and on the rising edges 324 and 334 of sync signals 221and 222, respectively, the lit camera 214-215 shutters are opened andthe dark camera 204-215 shutters are closed and the light panels areilluminated as shown by rising edge 344. The shutters remain in theirrespective states and the light panels remain illuminated for timeinterval 301. Then, on the rising edge 312 of sync signal 223 andfalling edges 322 and 332 of the sync signals 221 and 222, respectively,the lit camera 214-215 shutters are closed, the dark camera 204-215shutters are opened and the light panels are turned off as shown byfalling edge 342. The shutters and light panels are left in this statefor time interval 302. The process then repeats for each successiveframe time interval 303.

As a result, during the first time interval 301, a normally-lit image iscaptured by the color lit cameras 214-215, and the phosphorescent makeupis illuminated (and charged) with light from the light panels 208-209.During the second time interval 302, the light is turned off and thegrayscale dark cameras 204-205 capture an image of the glowingphosphorescent makeup on the performer. Because the light panels are offduring the second time interval 302, the contrast between thephosphorescent makeup and any surfaces in the room withoutphosphorescent makeup is extremely high (i.e., the rest of the room ispitch black or at least quite dark, and as a result there is nosignificant light reflecting off of surfaces in the room, other thanreflected light from the phosphorescent emissions), thereby improvingthe ability of the system to differentiate the various patterns appliedto the performer's face. In addition, because the light panels are onhalf of the time, the performer will be able to see around the roomduring the performance, and also the phosphorescent makeup is constantlyrecharged. The frequency of the synchronization signals is 1/(timeinterval 303) and may be set at such a high rate that the performer willnot even notice that the light panels are being turned on and off. Forexample, at a flashing rate of 90 Hz or above, virtually all humans areunable to perceive that a light is flashing and the light appears to becontinuously illuminated. In psychophysical parlance, when a highfrequency flashing light is perceived by humans to be continuouslyilluminated, it is said that “fusion” has been achieved. In oneembodiment, the light panels are cycled at 120 Hz; in anotherembodiment, the light panels are cycled at 140 Hz, both frequencies farabove the fusion threshold of any human. However, the underlyingprinciples of the invention are not limited to any particular frequency.

Surface Capture of Skin Using Phosphorescent Random Patterns

FIG. 4 shows images captured using the methods described above and the3D surface and textured 3D surface reconstructed from them. Prior tocapturing the images, a phosphorescent makeup was applied to a Caucasianmodel's face with an exfoliating sponge. Luminescent zinc sulfide with acopper activator (ZnS:Cu) is the phosphor responsible for the makeup'sphosphorescent properties. This particular formulation of luminescentZinc Sulfide is approved by the FDA color additives regulation 21 CFRPart 73 for makeup preparations. The particular brand is Fantasy F/XTTube Makeup; Product #: FFX; Color Designation: GL; manufactured byMehron Inc. of 100 Red Schoolhouse Rd. Chestnut Ridge, N.Y. 10977. Themotion capture session that produced these images utilized 8 grayscaledark cameras (such as cameras 204-205) surrounding the model's face froma plurality, of angles and 1 color lit camera (such as cameras 214-215)pointed at the model's face from an angle to provide the view seen inLit Image 401. The grayscale cameras were model A311 f from Basler A G,An der Strusbek 60-62, 22926 Ahrensburg, Germany, and the color camerawas a Basler model A311fc. The light panels 208-209 were flashed at arate of 72 flashes per second.

Lit Image 401 shows an image of the performer captured by one of thecolor lit cameras 214-215 during lit interval 301, when the light panels208-209 are on and the color lit camera 214-215 shutters are open. Notethat the phosphorescent makeup is quite visible on the performer's face,particularly the lips.

Dark Image 402 shows an image of the performer captured by one of thegrayscale dark cameras 204-205 during dark interval 302, when the lightpanels 208-209 are off and the grayscale dark camera 204-205 shuttersare open. Note that only random pattern of phosphorescent makeup isvisible on the surfaces where it is applied. All other surfaces in theimage, including the hair, eyes, teeth, ears and neck of the performerare completely black.

3D Surface 403 shows a rendered image of the surface reconstructed fromthe Dark Images 402 from grayscale dark cameras 204-205 (in thisexample, 8 grayscale dark cameras were used, each producing a singleDark Image 402 from a different angle) pointed at the model's face froma plurality of angles. One reconstruction process which may be used tocreate this image is detailed in co-pending application APPARATUS ANDMETHOD FOR PERFORMING MOTION CAPTURE USING A RANDOM PATTERN ON CAPTURESURFACES, Ser. No. 11/255,854, Filed Oct. 20, 2005. Note that 3D Surface403 was only reconstructed from surfaces where there was phosphorescentmakeup applied. Also, the particular embodiment of the technique thatwas used to produce the 3D Surface 403 fills in cavities in the 3Dsurface (e.g., the eyes and the mouth in this example) with a flatsurface.

Textured 3D Surface 404 shows the Lit Image 401 used as a texture mapand mapped onto 3D Surface 403 and rendered at an angle. AlthoughTextured 3D Surface 404 is a computer-generated 3D image of the model'sface, to the human eye it appears real enough that when it is renderedat an angle, such as it is in image 404, it creates the illusion thatthe model is turning her head and actually looking at an angle. Notethat no phosphorescent makeup was applied to the model's eyes and teeth,and the image of the eyes and teeth are mapped onto flat surfaces thatfill those cavities in the 3D surface. Nonetheless, the rest of the 3Dsurface is reconstructed so accurately, the resulting Textured 3DSurface 404 approaches photorealism. When this process is applied tocreate successive frames of Textured 3D Surfaces 404, when the framesare played back in real-time, the level of realism is such that, to theuntrained eye, the successive frames look like actual video of themodel, even though it is a computer-generated 3D image of the modelviewed from side angle.

Since the Textured 3D Surfaces 404 produces computer-generated 3Dimages, such computer-generated images can manipulated with far moreflexibility than actual video captured of the model. With actual videoit is often impractical (or impossible) to show the objects in the videofrom any camera angles other than the angle from which the video wasshot. With computer-generated 3D, the image can be rendered as if it isviewed from any camera angle. With actual video it is generallynecessary to use a green screen or blue screen to separate an objectfrom its background (e.g. so that a TV meteorologist can be compositedin front of a weather map), and then that green- or blue-screened objectcan only be presented from the point of view of the camera shooting theobject. With the technique just described, no green/blue screen isnecessary. Phosphorescent makeup, paint, or dye is applied to the areasdesired to be captured (e.g. the face, body and clothes of themeteorologist) and then the entire background will be separated from theobject. Further, the object can be presented from any camera angle. Forexample, the meteorologist can be shown from a straight-on shot, or froman side angle shot, but still composited in front of the weather map.

Further, a 3D generated image can be manipulated in 3D. For example,using standard 3D mesh manipulation tools (such as those in Maya, soldby Autodesk, Inc.) the nose can be shortened or lengthened, either forcosmetic reasons if the performer feels her nose would look better in adifferent size, or as a creature effect, to make the performer look likea fantasy character like Gollum of “Lord of the Rings.” More extensive3D manipulations could add wrinkles to the performers face to make herappear to be older, or smooth out wrinkles to make her look younger. Theface could also be manipulated to change the performer's expression, forExample, from a smile to a frown. Although some 2D manipulations arepossible with conventional 2D video capture, they are generally limitedto manipulations from the point of view of the camera. If the modelturns her head during the video sequence, the 2D manipulations appliedwhen the head is facing the camera would have to be changed when thehead is turned. 3D manipulations do not need to be changed, regardlessof which way the head is turned. As a result, the techniques describedabove for creating successive frames of Textured 3D Surface 404 in avideo sequence make it possible to capture objects that appear to looklike actual video, but nonetheless have the flexibility of manipulationas computer-generated 3D objects, offering enormous advantages inproduction of video, motion pictures, and also video games (wherecharacters may be manipulated by the player in 3D).

Note that in FIG. 4 the phosphorescent makeup is visible on the model'sface in Lit Image 401 and appears like a yellow powder has been spreadon her face. It is particularly prominent on her lower lip, where thelip color is almost entirely changed from red to yellow. Thesediscolorations appear in Textured 3D Surface 404, and they would be evenmore prominent on a dark-skinned model who is, for example, African inrace. Many applications (e.g. creating a fantasy 3D character likeGollum) only require 3D Surface 403, and Textured 3D Surface 404 wouldonly serve as a reference to the director of the motion capture sessionor as a reference to 3D animators manipulating the 3D Surface 403. Butin some applications, maintaining the actual skin color of the model'sskin is important and the discolorations from the phosphorescent makeupare not desirable.

Surface Capture Using Phosphorescent Makeup Mixed with Base

FIG. 5 shows a similar set of images as FIG. 4, captured and createdunder the same conditions: with 8 grayscale dark cameras (such as204-205), 1 color camera (such as 214-215), with the Lit Image 501captured by the color lit camera during the time interval when the LightArray 208-9 is on, and the Dark Image 502 captured by one of the 8grayscale dark cameras when the Light Array 208-9. 3D Surface 503 isreconstructed from the 8 Dark Images 502 from the 8 grayscale darkcameras, and Textured 3D Surface 504 is a rendering of the Lit Image 501texture-mapped onto 3D Surface 503 (and unlike image 404, image 504 isrendered from a camera angle similar to the camera angle of the colorlit camera that captured Lit Image 501).

However, there is a notable differences between the images of FIG. 5 andFIG. 4: The phosphorescent makeup that is noticeably visible in LitImage 401 and Textured 3D Surface 404 is almost invisible in Lit Image501 and Textured 3D Surface 504. The reason for this is that, ratherthan applying the phosphorescent makeup to the model in its pure form,as was done in the motion capture session of FIG. 4, in the embodimentillustrated in FIG. 5 the phosphorescent makeup was mixed with makeupbase and was then applied to the model. The makeup base used was “CleanMakeup” in “Buff Beige” color manufactured by Cover Girl, and it wasmixed with the same phosphorescent makeup used in the FIG. 4 shoot in aproportion of 80% phosphorescent makeup and 20% base makeup.

Note that mixing the phosphorescent makeup with makeup base does reducethe brightness of the phosphorescence during the Dark interval 302.Despite this, the phosphorescent brightness is still sufficient toproduce Dark Image 502, and there is enough dynamic range in the darkimages from the 8 grayscale dark cameras to reconstruct 3D Surface 503.As previously noted, some applications do not require an accuratecapture of the skin color of the model, and in that case it isadvantageous to not mix the phosphorescent makeup with base, and thenget the benefit of higher phosphorescent brightness during the Darkinterval 302 (e.g. higher brightness allows for a smaller aperturesetting on the camera lens, which allows for larger depth of field). Butsome applications do require an accurate capture of the skin color ofthe model. For such applications, it is advantageous to mix thephosphorescent makeup with base (in a color suited for the model's skintone) makeup, and work within the constraints of lower phosphorescentbrightness. Also, there are applications where some phosphor visibilityis acceptable, but not the level of visibility seen in Lit Image 401.For such applications, a middle ground can be found in terms of skincolor accuracy and phosphorescent brightness by mixing a higherpercentage of phosphorescent makeup relative to the base.

In another embodiment, luminescent zinc sulfide (ZnS:Cu) in its raw formis mixed with base makeup and applied to the model's face.

Surface Capture of Fabric with Phosphorescent Random Patterns

In another embodiment, the techniques described above are used tocapture cloth. FIG. 6 shows a capture of a piece of cloth (part of asilk pajama top) with the same phosphorescent makeup used in FIG. 4sponged onto it. The capture was done under the exact same conditionswith 8 grayscale dark cameras (such as 204-205) and 1 color lit camera(such as 214-215). The phosphorescent makeup can be seen slightlydiscoloring the surface of Lit Frame 601, during lit interval 301, butit can be seen phosphorescing brightly in Dark Frame 602, during darkinterval 302. From the 8 cameras of Dark Frame 602, 3D Surface 603 isreconstructed using the same techniques used for reconstructing the 3DSurfaces 403 and 503. And, then Lit Image 601 is texture-mapped onto 3DSurface 603 to produce Textured 3D Surface 604.

FIG. 6 shows a single frame of captured cloth, one of hundreds of framesthat were captured in a capture session while the cloth was moved,folded and unfolded. And in each frame, each area of the surface of thecloth was captured accurately, so long as at least 2 of the 8 grayscalecameras had a view of the area that was not overly oblique (e.g. thecamera optical axis was within 30 degrees of the area's surface normal).In some frames, the cloth was contorted such that there were areaswithin deep folds in the cloth (obstructing the light from the lightpanels 208-209), and in some frames the cloth was curved such that therewere areas that reflected back the light from the light panels 208-209so as to create a highlight (i.e. the silk fabric was shiny). Suchlighting conditions would make it difficult, if not impossible, toaccurately capture the surface of the cloth using reflected light duringlit interval 301 because shadow areas might be too dark for an accuratecapture (e.g. below the noise floor of the camera sensor) and somehighlights might be too bright for an accurate capture (e.g.oversaturating the sensor so that it reads the entire area as solidwhite). But, during the dark interval 302, such areas are readilycaptured accurately because the phosphorescent makeup emits light quiteuniformly, whether deep in a fold or on an external curve of the cloth.

Because the phosphor charges from any light incident upon it, includingdiffused or reflected light that is not directly from the light panels208-209, even phosphor within folds gets charged (unless the folds areso tightly sealed no light can get into them, but in such cases it isunlikely that the cameras can see into the folds anyway). Thisillustrates a significant advantage of utilizing phosphorescent makeup(or paint or dye) for creating patterns on (or infused within) surfacesto be captured: the phosphor is emissive and is not subject tohighlights and shadows, producing a highly uniform brightness level forthe patterns seen by the grayscale dark cameras 204-205, that neitherhas areas too dark nor areas too bright.

Another advantage of dyeing or painting a surface with phosphorescentdye or paint, respectively, rather than applying phosphorescent makeupto the surface is that with dye or paint the phosphorescent pattern onthe surface can be made permanent throughout a motion capture session.Makeup, by its nature, is designed to be removable, and a performer willnormally remove phosphorescent makeup at the end of a day's motioncapture shoot, and if not, almost certainly before going to bed.Frequently, motion capture sessions extend across several days, and as aresult, normally a fresh application of phosphorescent makeup is appliedto the performer each day prior to the motion capture shoot. Typically,each fresh application of phosphorescent makeup will result in adifferent random pattern. One of the techniques disclosed in co-pendingapplications is the tracking of vertices (“vertex tracking”) of thecaptured surfaces. Vertex tracking is accomplished by correlating randompatterns from one captured frame to the next. In this way, a point onthe captured surface can be followed from frame-to-frame. And, so longas the random patterns on the surface stay the same, a point on acaptured surface even can be tracked from shot-to-shot. In the case ofrandom patterns made using phosphorescent makeup, it is typicallypractical to leave the makeup largely undisturbed (although it ispossible for some areas to get smudged, the bulk of the makeup usuallystays unchanged until removed) during one day's-worth of motion captureshooting, but as previously mentioned it normally is removed at the endof the day. So, it is typically impractical to maintain the samephosphorescent random pattern (and with that, vertex tracking based ontracking a particular random pattern) from day-to-day. But when it comesto non-skin objects like fabric, phosphorescent dye or paint can be usedto create a random pattern. Because dye and paint are essentiallypermanent, random patterns will not get smudged during the motioncapture session, and the same random patterns will be unchanged fromday-to-day. This allows vertex tracking of dyed or painted objects withrandom patterns to track the same random pattern through the duration ofa multi-day motion capture session (or in fact, across multiple motioncapture sessions spread over long gaps in time if desired).

Skin is also subject to shadows and highlights when viewed withreflected light. There are many concave areas (e.g., eye sockets) thatoften are shadowed. Also, skin may be shiny and cause highlights, andeven if the skin is covered with makeup to reduce its shininess,performers may sweat during a physical performance, resulting inshininess from sweaty skin. Phosphorescent makeup emits uniformly bothfrom shiny and matte skin areas, and both from convex areas of the body(e.g. the nose bridge) and concavities (e.g. eye sockets). Sweat haslittle impact on the emission brightness of phosphorescent makeup.Phosphorescent makeup also charges while folded up in areas of the bodythat fold up (e.g. eyelids) and when it unfolds (e.g. when the performerblinks) the phosphorescent pattern emits light uniformly.

Returning back to FIG. 6, note that the phosphorescent makeup can beseen on the surface of the cloth in Lit Frame 601 and in Textured 3DSurface 604. Also, while this is not apparent in the images, although itmay be when the cloth is in motion, the phosphorescent makeup has asmall impact on the pliability of the silk fabric. In anotherembodiment, instead of using phosphorescent makeup (which of course isformulated for skin application) phosphorescent dye is used to createphosphorescent patterns on cloth. Phosphorescent dyes are available froma number of manufacturers. For example, it is common to find t-shirts atnovelty shops that have glow-in-the-dark patterns printed onto them withphosphorescent dyes. The dyes can also can be formulated manually bymixing phosphorescent powder (e.g. ZnS:Cu) with off-the-shelf clothingdyes, appropriate for the given type of fabric. For example, DharmaTrading Company with a store at 1604 Fourth Street, San Rafael, Calif.stocks a large number of dyes, each dye designed for certain fabricstypes (e.g. Dharma Fiber Reactive Procion Dye is for all natural fibers,Sennelier Tinfix Design—French Silk Dye is for silk and wool), as wellas the base chemicals to formulate such dyes. When phosphorescent powderis used as the pigment in such formulations, then a dye appropriate fora given fabric type is produced and the fabric can be dyed withphosphorescent pattern while minimizing the impact on the fabric'spliability.

Surface Capture of Stop-Motion Animation Characters with PhosphorescentRandom Patterns

In another embodiment, phosphor is embedded in silicone or a moldablematerial such as modeling clay in characters, props and background setsused for stop-motion animation. Stop-motion animation is a techniqueused in animated motion pictures and in motion picture special effects.An exemplary prior art stop-motion animation stage is illustrated inFIG. 7 a. Recent stop-motion animations are feature films Wallace &Gromit in The Curse of the Were-Rabbit (Academy Award-winning bestanimated feature film released in 2005) (hereafter referenced as WG) andCorpse Bride (Academy Award-nominated best animated feature filmreleased in 2005) (hereafter referred to as CB). Various techniques areused in stop-motion animation. In WG the characters 702-703 aretypically made of modeling clay, often wrapped around a metal armatureto give the character structural stability. In CB the characters 702-703are created from puppets with mechanical armatures which are thencovered with molded silicone (e.g. for a face), or some other material(e.g. for clothing). The characters 702-703 in both films are placed incomplex sets 701 (e.g. city streets, natural settings, or in buildings),the sets are lit with lights such as 708-709, a camera such as 705 isplaced in position, and then one frame is shot by the camera 705 (inmodern stop-motion animation, typically, a digital camera). Then thevarious characters (e.g. the man with a leash 702 and the dog 703) thatare in motion in the scene are moved very slightly. In the case of WB,often the movement is achieved by deforming the clay (and potentiallythe armature underneath it) or by changing a detailed part of acharacter 702-703 (e.g. for each frame swapping in a different mouthshape on a character 702-703 as it speaks). In the case of CB, oftenmotion is achieved by adjusting the character puppet 702-703 armature(e.g. a screwdriver inserted in a character puppet's 702-703 ear mightturn a screw that actuates the armature causing the character's 702-703mouth to open). Also, if the camera 705 is moving in the scene, then thecamera 705 is placed on a mechanism that allows it to be moved, and itis moved slightly each frame time. After all the characters 702-703 andthe camera 705 in a scene have been moved, another frame is captured bythe camera 705. This painstaking process continues frame-by-frame untilthe shot is completed.

There are many difficulties with the stop-motion animation process thatboth limit the expressive freedom of the animators, limit the degree ofrealism in motion, and add to the time and cost of production. One ofthese difficulties is animating many complex characters 702-703 within acomplex set 701 on a stop-motion animation stage such as that shown inFIG. 7 a. The animators often need to physically climb into the sets,taking meticulous care not to bump anything inadvertently, and then makeadjustments to character 702-703 expressions, often with sub-millimeterprecision. When characters 702-703 are very close to each other, it getseven more difficult. Also, sometimes characters 702-703 need to beplaced in a pose where a character 702-703 can easily fall over (e.g. acharacter 702-703 is doing a hand stand or a character 702-703 isflying). In these cases the character 702-703 requires some supportstructure that may be seen by the camera 705, and if so, needs to beerased from the shot in post-production.

In one embodiment illustrated by the stop-motion animation stage in FIG.7 b, phosphorescent phosphor (e.g. zinc sulfide) in powder form can bemixed (e.g. kneaded) into modeling clay resulting in the clay surfacephosphorescing in darkness with a random pattern. Zinc sulfide powderalso can be mixed into liquid silicone before the silicone is pouredinto a mold, and then when the silicone dries and solidifies, it haszinc sulfide distributed throughout. In another embodiment, zinc sulfidepowder can be spread onto the inner surface of a mold and then liquidsilicone can be poured into the mold to solidify (with the zinc sulfideembedded on the surface). In yet another embodiment, zinc sulfide ismixed in with paint that is applied to the surface of either modelingclay or silicone. In yet another embodiment, zinc sulfide is dyed intofabric worn by characters 702-703 or mixed into paint applied to propsor sets 701. In all of these embodiments the resulting effect is thatthe surfaces of the characters 702-703, props and sets 701 in the scenephosphoresce in darkness with random surface patterns.

At low concentrations of zinc sulfide in the various embodimentsdescribed above, the zinc sulfide is not significantly visible under thedesired scene illumination when light panels 208-208 are on. The exactpercentage of zinc sulfide depends on the particular material it ismixed with or applied to, the color of the material, and the lightingcircumstances of the character 702-703, prop or set 701. But,experimentally, the zinc sulfide concentration can be continuallyreduced until it is no longer visually noticeable in lighting situationswhere the character 702-703, prop or set 701 is to be used. This mayresult in a very low concentration of zinc sulfide and very lowphosphorescent emission. Although this normally would be a significantconcern with live action frame capture of dim phosphorescent patterns,with stop-motion animation, the dark frame capture shutter time can beextremely long (e.g. 1 second or more) because by definition, the sceneis not moving. With a long shutter time, even very dim phosphorescentemission can be captured accurately.

Once the characters 702-703, props and the set 701 in the scene are thusprepared, they look almost exactly as they otherwise would look underthe desired scene illumination when light panels 208-209 are on, butthey phosphoresce in random patterns when the light panels 208-209 areturned off. At this point all of the characters 702-703, props and theset 701 of the stop-motion animation can now be captured 3D using aconfiguration like that illustrated in FIGS. 2 a and 2 b and describedin the co-pending applications. (FIGS. 7 b-7 e illustrate stop-motionanimation stages with light panels 208-209, dark cameras 204-205 and litcameras 214-215 from FIGS. 2 a and 2 b surrounding the stop-motionanimation characters 702-703 and set 701. For clarity, the connectionsto devices 208-209, 204-205 and 214-215 have been omitted from FIGS. 7b-7 e, but in they would be hooked up as illustrated in FIGS. 2 a and 2b.) Dark cameras 204-205 and lit cameras 214-215 are placed around thescene illustrated in FIG. 7 b so as to capture whatever surfaces will beneeded to be seen in the final animation. And then, rather than rapidlyswitching sync signals 221-223 at a high capture frame rate (e.g. 90fps), the sync signals are switched very slowly, and in fact may beswitched by hand.

In one embodiment, the light panels 208-209 are left on while theanimators adjust the positions of the characters 702-703, props or anychanges to the set 701. Note that the light panels 208-209 could be anyillumination source, including incandescent lamps, because there is norequirement in stop-motion animation for rapidly turning on and off theillumination source. Once the characters 702-703, props and set 701 arein position for the next frame, lit cam sync signal 223 is triggered (bya falling edge transition in the presently preferred embodiment) and allof the lit cameras 214-215 capture a frame for a specified durationbased on the desired exposure time for the captured frames. In otherembodiments, different cameras may have different exposure times basedon individual exposure requirements.

Next, light panels 208-209 are turned off (either by sync signal 222 orby hand) and the lamps are allowed to decay until the scene is incomplete darkness (e.g. incandescent lamps may take many seconds todecay). Then, dark cam sync signal 221 is triggered (by a falling edgetransition in the presently preferred embodiment) and all of the darkcameras 208-209 capture a frame of the random phosphorescent patternsfor a specified duration based on the desired exposure time for thecaptured frames. Once again, different cameras have different exposuretimes based on individual exposure requirements. As previouslymentioned, in the case of very dim phosphorescent emissions, theexposure time may be quite long (e.g., a second or more). The upperlimit of exposure time is primarily limited by the noise accumulation ofthe camera sensors. The captured dark frames are processed by dataprocessing system 210 to produce 3D surface 207 and then to map theimages captured by the lit cameras 214-215 onto the 3D surface 207 tocreate textured 3D surface 217. Then, the light panels, 208-9 are turnedback on again, the characters 702-703, props and set 701 are movedagain, and the process described in this paragraph is repeated until theentire shot is completed.

The resulting output is the successive frames of textured 3D surfaces ofall of the characters 702-703, props and set 701 with areas of surfacesembedded or painted with phosphor that are in view of at least 2 darkcameras 204-205 at a non-oblique angle (e.g., <30 degrees from theoptical axis of a camera). When these successive frames are played backat the desired frame rate (e.g., 24 fps), the animated scene will cometo life, but unlike frames of a conventional stop-motion animation, theanimation will be able to be viewed from any camera position, just byrendering this textured 3D surfaces from a chosen camera position. Also,if the camera position of the final animation is to be in motion duringa frame sequence (e.g. if a camera is following a character 702-703), itis not necessary to have a physical camera moving in the scene. Rather,for each successive frame, the textured 3D surfaces of the scene aresimply rendered from the desired camera position for that frame, using a3D modeling/animation application software such as Maya (from Autodesk,Inc.).

In another embodiment, illustrated in FIGS. 7 c-7 e, some or all of thedifferent characters 702-703, props, and/or sets 701 within a singlestop-motion animation scene are shot separately, each in a configurationsuch as FIGS. 2 a and 2 b. For example, if a scene had man with leash702 and his dog 703 walking down a city street set 701, the city streetset 701, the man with leash 702, and the dog 703 would be shotindividually, each with separate motion capture systems as illustratedin FIG. 7 c (for city street set 701, FIG. 7 d (for man with leash 702)and FIG. 7 e (for dog 703)a. The stop-motion animation of the 2characters 702-703 and 1 set 701 would each then be separately capturedas individual textured 3D surfaces 217, in the manner described above.Then, with a 3D modeling and/or animation application software the 2characters 702-703 and 1 set 701 would be rendered together into a 3Dscene. In one embodiment, the light panel 208-209 lighting thecharacters 702-703 and the set 701 could be configured to be the same,so the man with leash 702 and the dog 703 appear to be illuminated inthe same environment as the set 701. In another embodiment, flatlighting (i.e. uniform lighting to minimize shadows and highlights) isused, and then lighting (including shadows and highlights) is simulatedby the 3D modeling/animation application software. Through the 3Dmodeling/animation application software the animators will be able tosee how the characters 702-703 look relative to each other and the set701, and will also be able to look at the characters 702-703 and set 701from any camera angle they wish, without having to move any of thephysical cameras 204-205 or 214-215 doing the capture.

This approach provides significant advantages to stop-motion animation.The following are some of the advantages of this approach: (a)individual characters 702-703 may be manipulated individually withoutworrying about the animator bumping into another character 702-703 orthe characters 702-703 bumping into each other, (b) the camera positionof the rendered frames may be chosen arbitrarily, including having thecamera position move in successive frames, (c) the rendered cameraposition can be one where it would not be physically possible to locatea camera 705 in a conventional stop-motion configuration (e.g. directlybetween 2 characters 702-703 that are close together, where there is noroom for a camera 705), (d) the lighting, including highlights andshadows can be controlled arbitrarily, including creating lightingsituations that are not physically possible to realize (e.g. making acharacter glow), (e) special effects can be applied to the characters702-703 (e.g. a ghost character 702-703 can be made translucent when itis rendered into the scene), (f) a character 702-703 can remain in aphysically stable position on the ground while in the scene it is not(e.g. a character 702-703 can be captured in an upright position, whileit is rendered into the scene upside down in a hand stand, or renderedinto the scene flying above the ground), (g) parts of the character702-703 can be held up by supports that do not have phosphor on them,and as such will not be captured (and will not have to be removed fromthe shot later in post-production), (h) detail elements of a character702-703, like mouth positions when the character 702-703 is speaking,can be rendered in by the 3D modeling/animation application, so they donot have be attached and then removed from the character 702-703 duringthe animation, (i) characters 702-703 can be rendered intocomputer-generated 3D scenes (e.g. the man with leash 702 and dog 703can be animated as clay animations, but the city street set 701 can be acomputer-generated scene), (j) 3D motion blur can be applied to theobjects as they move (or as the rendered camera position moves),resulting in a smoother perception of motion to the animation, and alsomaking possible faster motion without the perception of jitter.

Additional Phosphorescent Phosphors

In another embodiment, different phosphors other than ZnS:Cu are used aspigments with dyes for fabrics or other non-skin objects. ZnS:Cu is thepreferred phosphor to use for skin applications because it isFDA-approved as a cosmetic pigment. But a large variety of otherphosphors exist that, while not approved for use on the skin, are insome cases approved for use within materials handled by humans. One suchphosphor is SrAl₂O₄:Eu²⁺,Dy³⁺. Another is SrAl₂O₄:Eu²⁺. Both phosphorshave a much longer afterglow than ZnS:Cu for a given excitation.

Optimizing Phosphorescent Emission

Many phosphors that phosphoresce in visible light spectra are chargedmore efficiently by ultraviolet light than by visible light. This can beseen in chart 800 of FIG. 8 which show approximate excitation andemission curves of ZnS:Cu (which we shall refer to hereafter as “zincsulfide”) and various light sources. In the case of zinc sulfide, itsexcitation curve 811 spans from about 230 nm to 480 nm, with its peak ataround 360 nm. Once excited by energy in this range, its phosphorescencecurve 812 spans from about 420 nm to 650 nm, producing a greenish glow.The zinc sulfide phosphorescence brightness 812 is directly proportionalto the excitation energy 811 absorbed by the zinc sulfide. As can beseen by excitation curve 811, zinc sulfide is excited with varyingdegrees of efficiency depending on wavelength. For example, at a givenbrightness from an excitation source (i.e. in the case of the presentlypreferred embodiment, light energy from light panels 208-209) zincsulfide will absorb only 30% of the energy at 450 nm (blue light) thatit will absorb at 360 nm (UVA light, commonly called “black light”).Since it is desirable to get the maximum phosphorescent emission 812from the zinc sulfide (e.g. brighter phosphorescence will allow forsmaller lens apertures and longer depth of field), clearly it isadvantageous to excite the zinc sulfide with as much energy as possible.The light panels 208-209 can only produce up to a certain level of lightoutput before the light becomes uncomfortable for the performers. So, tomaximize the phosphorescent emission output of the zinc sulfide, ideallythe light panels 208-209 should output light at wavelengths that are themost efficient for exciting zinc sulfide.

Other phosphors that may be used for non-skin phosphorescent use (e.g.for dyeing fabrics) also are excited best by ultraviolet light. Forexample, SrAl₂O₄:Eu²⁺,Dy³⁺ and SrAl₂O₄:Eu²⁺ are both excited moreefficiently with ultraviolet light than visible light, and inparticular, are excited quite efficiently by UVA (black light).

As can be seen in FIG. 3, a requirement for a light source used for thelight panels 208-209 is that the light source can transition fromcompletely dark to fully lit very quickly (e.g. on the order of amillisecond or less) and from fully lit to dark very quickly (e.g. alsoon the order of a millisecond or less). Most LEDs fulfill thisrequirement quite well, typically turning on an off on the order ofmicroseconds. Unfortunately, though, current LEDs present a number ofissues for use in general lighting. For one thing, LEDs currentlyavailable have a maximum light output of approximately 35W. TheBL-43F0-0305 from Lamina Ceramics, 120 Hancock Lane, Westampton, N.J.08060 is one such RGB LED unit. For another, currently LEDs have specialpower supply requirements (in the case of the BL-43F0-0305, differentvoltage supplies are need for different color LEDs in the unit). Inaddition, current LEDs require very large and heavy heatsinks andproduce a great deal of heat. Each of these issues results in makingLEDs expensive and somewhat unwieldy for lighting an entire motioncapture stage for a performance. For example, if 3500 Watts were neededto light a stage, 100 35W LED units would be needed.

But, in addition to these disadvantages, the only very bright LEDscurrently available are white or RGB LEDs. In the case of both types ofLEDs, the wavelengths of light emitted by the LED does not overlap withwavelengths where the zinc sulfide is efficiently excited. For example,in FIG. 8 the emission curve 823 of the blue LEDs in the BL-43F0-0305LED unit is centered around 460 nm. It only overlaps with the tail endof the zinc sulfide excitation curve 811 (and the Red and Green LEDsdon't excite the zinc sulfide significantly at all). So, even if theblue LEDs are very bright (to the point where they are as bright as iscomfortable to the performer), only a small percentage of that lightenergy will excite the zinc sulfide, resulting in a relatively dimphosphorescence. Violet and UVA (“black light”) LEDs do exist, whichwould excite the zinc sulfide more efficiently, but they only currentlyare available at very low power levels, on the order of 0.1 Watts. Toachieve 3500 Watts of illumination would require 35,000 such 0.1 WattLEDs, which would be quite impractical and prohibitively expensive.

Fluorescent Lamps as a Flashing Illumination Source

Other lighting sources exist that output light at wavelengths that aremore efficiently absorbed by zinc sulfide. For example, fluorescentlamps (e.g. 482-S9 from Kino-Flo, Inc. 2840 North Hollywood Way,Burbank, Calif. 91505) are available that emit UVA (black light)centered around 350 nm with an emission curve similar to 821, andBlue/violet fluorescent lamps (e.g. 482-S10-S from Kino-Flo) exist thatemit bluish/violet light centered around 420 nm with an emission curvesimilar to 822. The emission curves 821 and 822 are much closer to thepeak of the zinc sulfide excitation curve 811, and as a result the lightenergy is far more efficiently absorbed, resulting in a much higherphosphorescent emission 812 for a given excitation brightness. Suchfluorescent bulbs are quite inexpensive (typically $15/bulb for a 48″bulb), produce very little heat, and are very light weight. They arealso available in high wattages. A typical 4-bulb fluorescent fixtureproduces 160 Watts or more. Also, theatrical fixtures are readilyavailable to hold such bulbs in place as staging lights. (Note that UVBand UVC fluorescent bulbs are also available, but UVB and UVC exposureis known to present health hazards under certain conditions, and as suchwould not be appropriate to use with human or animal performers withoutsuitable safety precautions.)

The primary issue with using fluorescent lamps is that they are notdesigned to switch on and off quickly. In fact, ballasts (the circuitsthat ignite and power fluorescent lamps) typically turn the lamps onvery slowly, and it is common knowledge that fluorescent lamps may takea second or two until they are fully illuminated.

FIG. 9 shows a diagrammatic view of a prior art fluorescent lamp. Theelements of the lamp are contained within a sealed glass bulb 910 which,in this example, is in the shape of a cylinder (commonly referred to asa “tube”). The bulb contains an inert gas 940, typically argon, and asmall amount of mercury 930. The inner surface of the bulb is coatedwith a phosphor 920. The lamp has 2 electrodes 905-906, each of which iscoupled to a ballast through connectors 901-904. When a large voltage isapplied across the electrodes 901-904, some of the mercury in the tubechanges from a liquid to a gas, creating mercury vapor, which, under theright electrical circumstances, emits ultraviolet light. The ultravioletlight excites the phosphor coating the inner surface of the bulb. Thephosphor then fluoresces light at a higher wavelength than theexcitation wavelength. A wide range of phosphors are available forfluorescent lamps with different wavelengths. For example, phosphorsthat are emissive at UVA wavelengths and all visible light wavelengthsare readily available off-the-shelf from many suppliers.

Standard fluorescent ballasts are not designed to switch fluorescentlamps on and off quickly, but it is possible to modify an existingballast so that it does. FIG. 10 is a circuit diagram of a prior art 27Watt fluorescent lamp ballast 1002 modified with an added sync controlcircuit 1001 of the present invention.

For the moment, consider only the prior art ballast circuit 1002 of FIG.10 without the modification 1001. Prior art ballast 1002 operates in thefollowing manner: A voltage doubler circuit converts 120VAC from thepower line into 300 volts DC. The voltage is connected to a half bridgeoscillator/driver circuit, which uses two NPN power transistors1004-1005. The half bridge driver, in conjunction with a multi-windingtransformer, forms an oscillator. Two of the transformer windingsprovide high drive current to the two power transistors 1004-1005. Athird winding of the transformer is in line with a resonant circuit, toprovide the needed feedback to maintain oscillation. The half bridgedriver generates a square-shaped waveform, which swings from +300 voltsduring one half cycle, to zero volts for the next half cycle. The squarewave signal is connected to an “LC” (i.e. inductor-capacitor) seriesresonant circuit. The frequency of the circuit is determined by theinductance Lres and the capacitance Cres. The fluorescent lamp 1003 isconnected across the resonant capacitor. The voltage induced across theresonant capacitor from the driver circuit provides the needed highvoltage AC to power the fluorescent lamp 1003. To kick the circuit intooscillation, the base of the power transistor 1005 is connected to asimple relaxation oscillator circuit. Current drawn from the 300v supplyis routed through a resistor and charges up a 0.1 uF capacitor. When thevoltage across the capacitor reaches about 20 volts, a DIAC (a bilateraltrigger diode) quickly switches and supplies power transistor 1005 witha current spike. This spike kicks the circuit into oscillation.

Synchronization control circuit 1001 is added to modify the prior artballast circuit 1002 described in the previous paragraph to allow rapidon-and-off control of the fluorescent lamp 1003 with a sync signal. Inthe illustrated embodiment in FIG. 10, a sync signal, such as syncsignal 222 from FIG. 2, is electrically coupled to the SYNC+ input.SYNC− is coupled to ground. Opto-isolator NEC PS2501-1 isolates theSYNC+ and SYNC− inputs from the high voltages in the circuit. Theopto-isolator integrated circuit consists of a light emitting diode(LED) and a phototransistor. The voltage differential between SYNC+ andSYNC− when the sync signal coupled to SYNC+ is at a high level (e.g.≧2.0V) causes the LED in the opto-isolator to illuminate and turn on thephototransistor in the opto-isolator. When this phototransistor isturned on, voltage is routed to the gate of an n-channel MOSFET Q1(Zetex Semiconductor ZVN4106F DMOS FET). MOSFET Q1 functions as a lowresistance switch, shorting out the base-emitter voltage of powertransistor 1005 to disrupt the oscillator, and turn off fluorescent lamp1003. To turn the fluorescent lamp back on, the sync signal (such as222) is brought to a low level (e.g. <0.8V), causing the LED in theopto-isolator to turn off, which turns off the opto-isolatorphototransistor, which turns off MOSFET Q1 so it no longer shorts outthe base-emitter voltage of power transistor 1005. This allows the kickstart circuit to initialize ballast oscillation, and the fluorescentlamp 1003 illuminates.

This process repeats as the sync signal coupled to SYNC+ oscillatesbetween high and low level. The synch control circuit 1001 combined withprior art ballast 1002 will switch fluorescent lamp 1003 on and offreliably, well in excess of 120 flashes per second. It should be notedthat the underlying principles of the invention are not limited to thespecific set of circuits illustrated in FIG. 10.

FIG. 11 shows the light output of fluorescent lamp 1003 when synchcontrol circuit 1001 is coupled to prior art ballast 1002 and a syncsignal 222 is coupled to circuit 1001 as described in the previousparagraph. Traces 1110 and 1120 are oscilloscope traces of the output ofa photodiode placed on the center of the bulb of a fluorescent lampusing the prior art ballast circuit 1002 modified with the sync controlcircuit 1001 of the present invention. The vertical axis indicates thebrightness of lamp 1003 and the horizontal axis is time. Trace 1110(with 2 milliseconds/division) shows the light output of fluorescentlamp 1003 when sync signal 222 is producing a 60 Hz square wave. Trace1120 (with the oscilloscope set to 1 millisecond/division and thevertical brightness scale reduced by 50%) shows the light output of lamp1003 under the same test conditions except now sync signal 222 isproducing a 250 Hz square wave. Note that the peak 1121 and minimum 1122(when lamp 1003 is off and is almost completely dark) are still bothrelatively flat, even at a much higher switching frequency. Thus, thesync control circuit 1001 modification to prior art ballast 1002produces dramatically different light output than the unmodified ballast1002, and makes it possible to achieve on and off switching offluorescent lamps at high frequencies as required by the motion capturesystem illustrated in FIG. 2 with timing similar to that of FIG. 3.

Although the modified circuit shown in FIG. 10 will switch a fluorescentlamp 1003 on and off rapidly enough for the requirements of a motioncapture system such as that illustrated in FIG. 2, there are certainproperties of fluorescent lamps that may be modified for use in apractical motion capture system.

FIG. 12 illustrates one of these properties. Traces 1210 and 1220 arethe oscilloscope traces of the light output of a General Electric Groand Sho fluorescent lamp 1003 placed in circuit 1002 modified by circuit1001, using a photodiode placed on the center of the bulb. Trace 1210shows the light output at 1 millisecond/division, and Trace 1220 showsthe light output at 20 microseconds/division. The portion of thewaveform shown in Trace 1220 is roughly the same as the dashed line area1213 of Trace 1210. Sync signal 222 is coupled to circuit 1002 asdescribed previously and is producing a square wave at 250 Hz. Peaklevel 1211 shows the light output when lamp 1003 is on and minimum 1212shows the light output when lamp 1003 is off. While Trace 1210 shows thepeak level 1211 and minimum 1212 as fairly flat, upon closer inspectionwith Trace 1220, it can be seen that when the lamp 1003 is turned off,it does not transition from fully on to completely off instantly.Rather, there is a decay curve of approximately 200 microseconds (0.2milliseconds) in duration. This is apparently due to the decay curie ofthe phosphor coating the inside of the fluorescent bulb (i.e. when thelamp 1003 is turned off, the phosphor continues to fluoresce for a briefperiod of time). So, when sync signal 222 turns off the modified ballast1001-1002, unlike LED lights which typically switch off within amicrosecond, fluorescent lamps take a short interval of time until theydecay and become dark.

There exists a wide range of decay periods for different brands andtypes of fluorescent lamps, from as short as 200 microseconds, to aslong as over a millisecond. To address this property of fluorescentlamps, one embodiment of the invention adjusts signals 221-223. Thisembodiment will be discussed shortly.

Another property of fluorescent lamps that impacts their usability witha motion capture system such as that illustrated in FIG. 2 is that theelectrodes within the bulb are effectively incandescent filaments thatglow when they carry current through them, and like incandescentfilaments, they continue to glow for a long time (often a second ormore) after current is removed from them. So, even if they are switchedon and off rapidly (e.g. at 90 Hz) by sync signal 222 using ballast 1002modified by circuit 1001, they continue to glow for the entire darkinterval 302. Although the light emitted from the fluorescent bulb fromthe glowing electrodes is very dim relative to the fully illuminatedfluorescent bulb, it is still is a significant amount of light, and whenmany fluorescent bulbs are in use at once, together the electrodes addup to a significant amount of light contamination during the darkinterval 302, where it is advantageous for the room to be as dark aspossible.

FIG. 13 illustrates one embodiment of the invention which addresses thisproblem. Prior art fluorescent lamp 1350 is shown in a state 10milliseconds after the lamp as been shut off. The mercury vapor withinthe lamp is no longer emitting ultraviolet light and the phosphor liningthe inner surface of the bulb is no longer emitting a significant amountof light. But the electrodes 1351-1352 are still glowing because theyare still hot. This electrode glowing results in illuminated regions1361-1362 near the ends of the bulb of fluorescent lamp 1350.

Fluorescent lamp 1370 is a lamp in the same state as prior art lamp1350, 10 milliseconds after the bulb 1370 has been shut off, with itselectrodes 1371-1372 still glowing and producing illuminated regions1381-1382 near the ends of the bulb of fluorescent lamp 1370, but unlikeprior art lamp 1350, wrapped around the ends of lamp 1370 is opaque tape1391 and 1392 (shown as see-through with slanted lines for the sake ofillustration). In the presently preferred embodiment black gaffers' tapeis used, such as 4″ P-665 from Permacel, A Nitto Denko Company, USHighway No. 1, P.O. Box 671, New Brunswick, N.J. 08903. The opaque tape1391-1392 serves to block almost all of the light from glowingelectrodes 1371-1372 while blocking only a small amount of the overalllight output of the fluorescent lamp when the lamp is on during litinterval 301. This allows the fluorescent lamp to become much darkerduring dark interval 302 when being flashed on and off at a high rate(e.g. 90 Hz). Other techniques can be used to block the light from theglowing electrodes, including other types of opaque tape, painting theends of the bulb with an opaque paint, or using an opaque material (e.g.sheets of black metal) on the light fixtures holding the fluorescentlamps so as to block the light emission from the parts of thefluorescent lamps containing electrodes.

Returning now to the light decay property of fluorescent lampsillustrated in FIG. 12, if fluorescent lamps are used for light panels208-209, the synchronization signal timing shown in FIG. 3 will notproduce optimal results because when Light Panel sync signal 222 dropsto a low level on edge 332, the fluorescent light panels 208-209 willtake time to become completely dark (i.e. edge 342 will gradually dropto dark level). If the Dark Cam Sync Signal triggers the grayscalecameras 204-205 to open their shutters at the same time as edge 322, thegrayscale camera will capture some of the scene lit by the afterglow oflight panels 208-209 during its decay interval. Clearly, FIG. 3's timingsignals and light output behavior is more suited for light panels208-209 using a lighting source like LEDs that have a much faster decaythan fluorescent lamps.

Synchronization Timing for Fluorescent Lamps

FIG. 14 shows timing signals which are better suited for use withfluorescent lamps and the resulting light panel 208-209 behavior (notethat the duration of the decay curve 1442 is exaggerated in this andsubsequent timing diagrams for illustrative purposes). The rising edge1434 of sync signal 222 is roughly coincident with rising edge 1414 oflit cam sync signal 223 (which opens the lit camera 214-215 shutters)and with falling edge 1424 of dark cam sync signal 223 (which closes thedark camera 204-205 shutters). It also causes the fluorescent lamps inthe light panels 208-209 to illuminate quickly. During lit time interval1401, the lit cameras 214-215 capture a color image illuminated by thefluorescent lamps, which are emitting relatively steady light as shownby light output level 1443.

At the end of lit time interval 1401, the falling edge 1432 of syncsignal 222 turns off light panels 208-209 and is roughly coincident withthe rising edge 1412 of lit cam sync signal 223, which closes theshutters of the lit cameras 214-215. Note, however, that the lightoutput of the light panels 208-209 does not drop from lit to darkimmediately, but rather slowly drops to dark as the fluorescent lampphosphor decays as shown by edge 1442. When the light level of thefluorescent lamps finally reaches; dark level 1441, dark cam sync signal221 is dropped from high to low as shown by edge 1422, and this opensthe shutters of dark cameras 204-205. This way the dark cameras 204-205only capture the emissions from the phosphorescent makeup, paint or dye,and do not capture the reflection of light from any objects illuminatedby the fluorescent lamps during the decay interval 1442. So, in thisembodiment the dark interval 1402 is shorter than the lit interval 1401,and the dark camera 204-205 shutters are open for a shorter period oftime than the lit camera 214-205 shutters.

Another embodiment is illustrated in FIG. 15 where the dark interval1502 is longer than the lit interval 1501. The advantage of thisembodiment is it allows for a longer shutter time for the dark cameras204-205. In this embodiment, light panel sync signal 222 falling edge1532 occurs earlier which causes the light panels 208-209 to turn off.Lit cam sync signal 223 rising edge 1512 occurs roughly coincident withfalling edge 1532 and closes the shutters on the lit cameras 214-5. Thelight output from the light panel 208-209 fluorescent lamps begins todecay as shown by edges 1542 and finally reaches dark level 1541. Atthis point dark cam sync signal 221 is transitions to a low state onedge 1522, and the dark cameras 204-205 open their shutters and capturethe phosphorescent emissions.

Note that in the embodiments shown in both FIGS. 14 and 15 the litcamera 214-215 shutters were only open while the light output of thelight panel 208-209 fluorescent lamps was at maximum. In anotherembodiment, the lit camera 214-215 shutters can be open during theentire time the fluorescent lamps are emitting any light, so as tomaximize the amount of light captured. In this situation, however, thephosphorescent makeup, paint or dye in the scene will become moreprominent relative to the non-phosphorescent areas in the scene becausethe phosphorescent areas will continue to emit light fairly steadilyduring the fluorescent lamp decay while the non-phosphorescent areaswill steadily get darker. The lit cameras 214-215 will integrate thislight during the entire time their shutters are open.

In yet another embodiment the lit cameras 214-215 leave their shuttersopen for some or all of the dark time interval 1502. In this case, thephosphorescent areas in the scene will appear very prominently relativeto the non-phosphorescent areas since the lit cameras 214-215 willintegrate the light during the dark time interval 1502 with the lightfrom the lit time interval 1501.

Because fluorescent lamps are generally not sold with specificationsdetailing their phosphor decay characteristics, it is necessary todetermine the decay characteristics of fluorescent lamps experimentally.This can be readily done by adjusting the falling edge 1522 of syncsignal 221 relative to the falling edge 1532 of sync signal 222, andthen observing the output of the dark cameras 204-205. For example, inthe embodiment shown in FIG. 15, if edge 1522 falls too soon after edge1532 during the fluorescent light decay 1542, then non-phosphorescentobjects will be captured in the dark cameras 204-205. If the edge 1522is then slowly delayed relative to edge 1532, the non-phosphorescentobjects in dark camera 204-205 will gradually get darker until theentire image captured is dark, except for the phosphorescent objects inthe image. At that point, edge 1522 will be past the decay interval 1542of the fluorescent lamps. The process described in this paragraph can bereadily implemented in an application on a general-purpose computer thatcontrols the output levels of sync signals 221-223.

In another embodiment the decay of the phosphor in the fluorescent lampsis such that even after edge 1532 is delayed as long as possible after1522 to allow for the dark cameras 204-205 to have a long enough shuttertime to capture a bright enough image of phosphorescent patterns in thescene, there is still a small amount of light from the fluorescent lampilluminating the scene such that non-phosphorescent objects in the sceneare slightly visible. Generally, this does not present a problem for thepattern processing techniques described in the co-pending applicationsidentified above. So long as the phosphorescent patterns in the sceneare substantially brighter than the dimly-lit non-fluorescent objects inthe scene, the pattern processing techniques will be able to adequatelycorrelate and process the phosphorescent patterns and treat the dimlylit non-fluorescent objects as noise.

Synchronizing Cameras with Lower Frame Rates than the Light PanelFlashing Rate

In another embodiment the lit cameras 214-215 and dark cameras 204-205are operated at a lower frame rate than the flashing rate of the lightpanels 208-209. For example, the capture frame rate may be 30 frames persecond (fps), but so as to keep the flashing of the light panels 208-209about the threshold of human perception, the light panels 208-209 areflashed at 90 flashes per second. This situation is illustrated in FIG.16. The sync signals 221-3 are controlled the same as the are in FIG. 15for lit time interval 1601 and dark time interval 1602 (light cycle 0),but after that, only light panel 208-9 sync signal 222 continues tooscillate for light cycles 1 and 2. Sync signals 221 and 223 remain inconstant high state 1611 and 1626 during this interval. Then duringlight cycle 3, sync signals 221 and 223 once again trigger with edges1654 and 1662, opening the shutters of lit cameras 214-215 during littime interval 1604, and then opening the shutters of dark cameras204-205 during dark time interval 1605.

In another embodiment where the lit cameras 214-215 and dark cameras204-205 are operated at a lower frame rate than the flashing rate of thelight panels 208-209, sync signal 223 causes the lit cameras 214-215 toopen their shutters after sync signal 221 causes the dark cameras204-205 to open their shutters. This is illustrated in FIG. 17. Anadvantage of this timing arrangement over that of FIG. 16 is thefluorescent lamps transition from dark to lit (edge 1744) more quicklythan they decay from lit to dark (edge 1742). This makes it possible toabut the dark frame interval 1702 more closely to the lit frame interval1701. Since captured lit textures are often used to be mapped onto 3Dsurfaces reconstructed from dark camera images, the closer the lit anddark captures occur in time, the closer the alignment will be if thecaptured object is in motion.

In another embodiment where the lit cameras 214-215 and dark cameras204-205 are operated at a lower frame rate than the flashing rate of thelight panels 208-209, the light panels 208-209 are flashed with varyinglight cycle intervals so as to allow for longer shutter times for eitherthe dark cameras 204-205 or lit cameras 214-215, or to allow for longershutters times for both cameras. An example of this embodiment isillustrated in FIG. 18 where the light panels 208-209 are flashed at 3times the frame rate of cameras 204-205 and 214-215, but the openshutter interval 1821 of the dark cameras 204-205 is equal to almosthalf of the entire frame time 1803. This is accomplished by having lightpanel 208-209 sync signal 222 turn off the light panels 208-209 for along dark interval 1802 while dark cam sync signal 221 opens the darkshutter for the duration of long dark interval 1802. Then sync signal222 turns the light panels 208-209 on for a brief lit interval 1801, tocomplete light cycle 0 and then rapidly flashes the light panels 208-209through light cycles 1 and 2. This results in the same number of flashesper second as the embodiment illustrated in FIG. 17, despite the muchlonger dark interval 1802. The reason this is a useful configuration isthat the human visual system will still perceive rapidly flashing lights(e.g. at 90 flashes per second) as being lit continuously, even if thereare some irregularities to the flashing cycle times. By varying theduration of the lit and dark intervals of the light panels 208-209, theshutter times of either the dark cameras 204-205, lit cameras 214-215 orboth can be lengthened or shortened, while still maintaining the humanperception that light panels 208-209 are continuously lit.

High Aggregate Frame Rates from Cascaded Cameras

FIG. 19 illustrates another embodiment where lit cameras 1941-1946 anddark cameras 1931-1936 are operated at a lower frame rate than theflashing rate of the light panels 208-209. FIG. 19 illustrates a similarmotion capture system configuration as FIG. 2 a, but given spacelimitations in the diagram only the light panels, the cameras, and thesynchronization subsystem is shown. The remaining components of FIG. 2 athat are not shown (i.e. the interfaces from the cameras to their cameracontrollers and the data processing subsystem, as well as the output ofthe data processing subsystem) are a part of the full configuration thatis partially shown in FIG. 19, and they are coupled to the components ofFIG. 19 in the same manner as they are to the components of FIG. 2 a.Also, FIG. 19 shows the Light Panels 208-209 in their “lit” state. LightPanels 208-209 can be switched off by sync signal 222 to their “dark”state, in which case performer 202 would no longer be lit and only thephosphorescent pattern applied to her face would be visible, as it isshown in FIG. 2 b.

FIG. 19 shows 6 lit cameras 1941-1946 and 6 dark cameras 1931-1936. Inthe presently preferred embodiment color cameras are used for the litcameras 1941-1946 and grayscale cameras are used for the dark camera1931-1936, but either type could be used for either purpose. Theshutters on the cameras 1941-1946 and 1931-1936 are driven by syncsignals 1921-1926 from sync generator PCI card 224. The sync generatorcard is installed in sync generator PC 220, and operates as previouslydescribed. (Also, in another embodiment it may be replaced by using theparallel port outputs of sync generator PC 220 to drive sync signals1921-1926, and in this case, for example, bit 0 of the parallel portwould drive sync signal 222, and bits 1-6 of the parallel port woulddrive sync signals 1921-1926, respectively.)

Unlike the previously described embodiments, where there is one syncsignal 221 for the dark cameras and one sync signal 223 for the litcameras, in the embodiment illustrated in FIG. 19, there are 3 syncsignals 1921-1923 for the dark cameras and 3 sync signals 1924-1926 forthe dark cameras. The timing for these sync signals 1921-1926 is shownin FIG. 20. When the sync signals 1921-1926 are in a high state theycauses the shutters of the cameras attached to them to be closed, whenthe sync signals are in a low state, they cause the shutters of thecameras attached to them to be open.

In this embodiment, as shown in FIG. 20, the light panels 208-209 areflashed at a uniform 90 flashes per second, as controlled by sync signal222. The light output of the light panels 208-209 is also shown,including the fluorescent lamp decay 2042. Each camera 1931-1936 and1941-1946 captures images at 30 frames per second (fps), exactly at a1:3 ratio with the 90 flashes per second rate of the light panels. Eachcamera captures one image per each 3 flashes of the light panels, andtheir shutters are sequenced in a “cascading” order, as illustrated inFIG. 20. A sequence of 3 frames is captured in the following manner:

Sync signal 222 transitions with edge 2032 from a high to low state2031. Low state 2031 turns off light panels 208-209, which graduallydecay to a dark state 2041 following decay curve 2042. When the lightpanels are sufficiently dark for the purposes of providing enoughcontrast to separate the phosphorescent makeup, paint, or dye from thenon-phosphorescent surfaces in the scene, sync signal 1921 transitionsto low state 2021. This causes dark cameras 1931-1932 to open theirshutters and capture a dark frame. After the time interval 2002, syncsignal 222 transitions with edge 2034 to high state 2033 which causesthe light panels 208-209 to transition with edge 2044 to lit state 2043.Just prior to light panels 208-209 becoming lit, sync signal 1921transitions to high state 2051 closing the shutter of dark cameras1931-1932. Just after the light panels 208-209 become lit, sync signal1924 transition to low state 2024, causing the shutters on the litcameras 1941-1942 to open during time interval 2001 and capture a litframe. Sync signal 222 transitions to a low state, which turns off thelight panels 208-9, and sync signal 1924 transitions to a high state atthe end of time interval 2001, which closes the shutters on lit cameras1941-1942.

The sequence of events described in the preceding paragraphs repeats 2more times, but during these repetitions sync signals 1921 and 1924remain high, keeping their cameras shutters closed. For the firstrepetition, sync signal 1922 opens the shutter of dark cameras 1933-1934while light panels 208-209 are dark and sync signal 1925 opens theshutter of lit cameras 1943-1944 while light panels 208-209 are lit. Forthe second repetition, sync signal 1923 opens the shutter of darkcameras 1935-1936 while light panels 208-209 are dark and sync signal1926 opens the shutter of lit cameras 1945-1946 while light panels208-209 are lit.

Then, the sequence of events described in the prior 2 paragraphscontinues to repeat while the motion capture session illustrated in FIG.19 is in progress, and thus a “cascading” sequence of camera capturesallows 3 sets of dark and 3 sets of lit cameras to capture motion at 90fps (i.e. equal to the light panel flashing rate of 90 flashes persecond), despite the fact each cameras is only capturing images at 30fps. Because each camera only captures 1 of every 3 frames, the capturedframes stored by the data processing system 210 are then interleaved sothat the stored frame sequence at 90 fps has the frames in proper orderin time. After that interleaving operation is complete, the dataprocessing system will output reconstructed 3D surfaces 207 and textured3D surfaces 217 at 90 fps.

Although the “cascading” timing sequence illustrated in FIG. 20 willallow cameras to operate at 30 fps while capturing images at anaggregate rate of 90 fps, it may be desirable to be able to switch thetiming to sometimes operate all of the cameras 1921-1923 and 1924-1926synchronously. An example of such a situation is for the determinationof the relative position of the cameras relative to each other. Preciseknowledge of the relative positions of the dark cameras 1921-1923 isused for accurate triangulation between the cameras, and preciseknowledge of the position of the lit cameras 1924-1926 relative to thedark cameras 1921-1923 is used for establishing how to map the texturemaps captured by the lit cameras 1924-1926 onto the geometryreconstructed from the images captured by the dark cameras 1921-1923.One prior art method (e.g. that is used to calibrate cameras for themotion capture cameras from Motion Analysis Corporation) to determinethe relative position of fixed cameras is to place a known object (e.g.spheres on the ends of a rods in a rigid array) within the field of viewof the cameras, and then synchronously (i.e. with the shutters of allcameras opening and closing simultaneously) capture successive frames ofthe image of that known object by all the cameras as the object is inmotion. By processing successive frames from all of the cameras, it ispossible to calculate the relative position of the cameras to eachother. But for this method to work, all of the cameras need to besynchronized so that they capture images simultaneously. If the camerashutters do not open simultaneously, then when each non-simultaneousshutter opens, its camera will capture the moving object at a differentposition in space than other cameras whose shutters open at differenttimes. This will make it more difficult (or impossible) to preciselydetermine the relative position of all the cameras to each other.

FIG. 21 illustrates in another embodiment how the sync signals 1921-6can be adjusted so that all of the cameras 1931-1936 and 1941-1946 opentheir shutters simultaneously. Sync signals 1921-1926 all transition tolow states 2121-2126 during dark time interval 2102. Although the lightpanels 208-209 would be flashed 90 flashes a second, the cameras wouldbe capturing frames synchronously to each other at 30 fps. (Note that inthis case, the lit cameras 1941-1946 which, in the presently preferredembodiment are color cameras, also would be capturing frames during thedark interval 2102 simultaneously with the dark cameras 1931-1936.)Typically, this synchronized mode of operation would be done when acalibration object (e.g. an array of phosphorescent spheres) was placedwithin the field of view of some or all of the cameras, and potentiallymoved through successive frames, usually before or after a motioncapture of a performer. In this way, the relative position of thecameras could determined while the cameras are running synchronously at30 fps, as shown in FIG. 21. Then, the camera timing would be switchedto the “cascading” timing shown in FIG. 20 to capture a performance at90 fps. When the 90 fps frames are reconstructed by data processingsystem 210, then camera position information, determined previously (orsubsequently) to the 90 fps capture with the synchronous mode time shownin FIG. 21, will be used to both calculate the 3D surface 207 and mapthe captured lit frame textures onto the 3D surface to create textured3D surface 217

When a scene is shot conventionally using prior art methods and camerasare capturing only 2D images of that scene, the “cascading” technique touse multiple slower frame rate cameras to achieve a higher aggregateframe rate as illustrated in FIGS. 19 and 20 will not producehigh-quality results. The reason for this is each camera in a “cascade”(e.g. cameras 1931, 1933 and 1935) will be viewing the scene from adifferent point of view. If the captured 30 fps frames of each cameraare interleaved together to create a 90 fps sequence of successiveframes in time, then when the 90 fps sequence is viewed, it will appearto jitter, as if the camera was rapidly jumping amongst multiplepositions. But when slower frame rate cameras are “cascaded” to achievea higher aggregate frame rate as illustrate in FIGS. 19 and 20 for thepurpose capturing the 3D surfaces of objects in a scene, as describedherein and in combination with the methods described in the co-pendingapplications, the resulting 90 fps interleaved 3D surfaces 207 andtextured 3D surfaces 217 do not exhibit jitter at all, but rather lookcompletely stable. The reason is the particular position of the cameras1931-1936 and 1941-1946 does not matter in the reconstruction 3Dsurfaces, just so long as the at least a pair of dark cameras 1931-1936during each dark frame interval 2002 has a non-oblique view (e.g. <30degrees) of the surface area (with phosphorescent makeup, paint or dye)to be reconstructed. This provides a significant advantage overconventional prior art 2D motion image capture (i.e. commonly known asvideo capture), because typically the highest resolution sensorscommercially available at a given time have a lower frame rate thancommercially available lower resolution sensors. So, 2D motion imagecapture at high resolutions is limited to the frame rate of a singlehigh resolution sensor. A 3D motion surface capture at high resolution,under the principles described herein, is able to achieve n times theframes rate of a single high resolution sensor, where n is the number ofcamera groups “cascaded” together, per the methods illustrated in FIGS.19 and 20.

Color Mapping of Phosphor Brightness

Ideally, the full dynamic range, but not more, of dark cameras 204-205should be utilized to achieve the highest quality pattern capture. Forexample, if a pattern is captured that is too dark, noise patterns inthe sensors in cameras 204-205 may become as prominent as capturedpatterns, resulting in incorrect 3D reconstruction. If a pattern is toobright, some areas of the pattern may exceed the dynamic range of thesensor, and all pixels in such areas will be recorded at the maximumbrightness level (e.g. 255 in an 8-bit sensor), rather than at thevariety or brightness levels that actually make up that area of thepattern. This also will result in incorrect 3D reconstruction. So, priorto capturing a pattern, per the techniques described herein, it isadvantageous to try to make sure the brightness of the patternthroughout is not too dark, nor too bright (e.g. not reaching themaximum brightness level of the camera sensor).

When phosphorescent makeup is applied to a performer, or whenphosphorescent makeup, paint or dye is applied to an object, it isdifficult for the human eye to evaluate whether the phosphor applicationresults in a pattern captured by the dark cameras 204-205 that is brightenough in all locations or too bright in some locations. FIG. 22 image2201 shows a cylinder covered in a random pattern of phosphor. It isdifficult, when viewing this image on a computer display (e.g. an LCDmonitor) to determine precisely if there are parts of the pattern thatare too bright (e.g. location 2220) or too dark (e.g. location 2210).There are many reasons for this. Computer monitors often do not have thesame dynamic range as a sensor (e.g. a computer monitor may only display128 unique gray levels, while the sensor captures 256 gray levels). Thebrightness and/or contrast may not be set correctly on the monitor.Also, the human eye may have trouble determining what constitutes amaximum brightness level because the brain may adapt to the brightnessit sees, and consider whatever is the brightest area on the screen to bethe maximum brightness. For all of these reasons, it is helpful to havean objective measure of brightness that humans can readily evaluate whenapplying phosphorescent makeup, paint or dye. Also, it is helpful tohave an objective measure brightness as the lens aperture and/or gain isadjusted on dark cameras 204-205 and/or the brightness of the lightpanels 208-209 is adjusted.

Image 2202 shows such an objective measure. It shows the same cylinderas image 2201, but instead of showing the brightness of each pixel ofthe image as a grayscale level (in this example, from 0 to 255), itshows it as a color. Each color represents a range of brightness. Forexample, in image 2202 blue represents brightness ranges 0-32, orangerepresents brightness ranges 192-223 and dark red represents brightnessranges 224-255. Other colors represent other brightness ranges. Area2211, which is blue, is now clearly identifiable as an area that is verydark, and area 2221, which is dark red, is now clearly identifiable asan area that is very bright. These determinations can be readily made bythe human eye, even if the dynamic range of the display monitor is lessthan that of the sensor, or if the display monitor is incorrectlyadjusted, or if the brain of the observer adapts to the brightness ofthe display. With this information the human observer can change theapplication of phosphorescent makeup, dye or paint. The human observercan also adjust the aperture and/or the gain setting on the cameras204-205 and/or the brightness of the light panels 208-209.

In one embodiment image 2202 is created by application software runningon one camera controller computer 225 and is displayed on a color LCDmonitor attached to the camera controller computer 225. The cameracontroller computer 225 captures a frame from a dark camera 204 andplaces the pixel values of the captured frame in an array in its RAM.For example, if the dark cameras 204 is a 640×480 grayscale camera with8 bits/pixel, then the array would be a 640×480 array of 8-bit bytes inRAM. Then, the application takes each pixel value in the array and usesit as an index into a lookup table of colors, with as many entries asthe number of possible pixel values. With 8 bits/pixel, the lookup tablehas 256 entries. Each of the entries in the lookup table is pre-loaded(by the user or the developer of the application) with the desired Red,Green, Blue (RGB) color value to be displayed for the given brightnesslevel. Each brightness level may be given a unique color, or a range ofbrightness levels can share a unique color. For example, for image 2202,lookup table entries 0-31 are all loaded with the RGB value for blue,entries 192-223 are loaded with the RGB value for orange and entries224-255 are loaded with the RGB value for dark red. Other entries areloaded with different RGB color values. The application uses each pixelvalue from the array (e.g. 640×480 of 8-bit grayscale values) of thecaptured frame as an index into this color lookup take, and forms a newarray (e.g. 640×480 of 24-bit RGB values) of the looked-up colors. Thisnew array of look-up colors is then displayed, producing a color imagesuch as 1102.

If a color camera (either lit camera 214 or dark camera 204) is used tocapture the image to generate an image such as 2202, then one step isfirst performed after the image is captured find before it is processedas described in the preceding paragraph. The captured RGB output of thecamera is stored in an array in camera controller computer 225 RAM (e.g.640×480 with 24 bits/pixel). The application running on cameracontroller computer 225 then calculates the average brightness of eachpixel by averaging the Red, Green and Blue values of each pixel (i.e.Average=(R+G+B)/3), and places those averages in a new array (e.g.640×480 with 8 bits/pixel). This array of Average pixel brightnesses(the “Average array”) will soon be processed as if it were the pixeloutput of a grayscale camera, as described in the prior paragraph, toproduce a color image such as 2202. But, first there is one more step:the application examines each pixel in the captured RGB array to see ifany color channel of the pixel (i.e. R, G, or B) is at a maximumbrightness value (e.g. 255). If any channel is, then the applicationsets the value in the Average array for that pixel to the maximumbrightness value (e.g. 255). The reason for this is that it is possiblefor one color channel of a pixel to be driven beyond maximum brightness(but only output a maximum brightness value), while the other colorchannels are driven by relatively dim brightness. This may result in anaverage calculated brightness for that pixel that is a middle-rangelevel (and would not be considered to be a problem for good-qualitypattern capture). But, if any of the color channels has been overdrivenin a given pixel, then that will result in an incorrect pattern capture.So, by setting the pixel value in the Average array to maximumbrightness, this produces a color images 2202 where that pixel is shownto be at the highest brightness, which would alert a human observer ofimage 1102 of the potential of a problem for a high-quality patterncapture.

It should be noted that the underlying principles of the invention arenot limited to the specific color ranges and color choices illustratedin FIG. 22. Also, other methodologies can be used to determine thecolors in 2202, instead of using only a single color lookup table. Forexample, in one embodiment the pixel brightness (or average brightness)values of a captured image is used to specify the hue of the colordisplayed. In another embodiment, a fixed number of lower bits (e.g. 4)of the pixel brightness (or average brightness) values of a capturedimage are set to zeros, and then the resulting numbers are used tospecify the hue for each pixel. This has the effect of assigning eachsingle hue to a range of brightnesses.

Surface Reconstruction from Multiple Range Data Sets

Correlating lines or random patterns captured by one camera with imagesfrom other cameras as described above provides range information foreach camera. In one embodiment of the invention, range information frommultiple cameras is combined in three steps: (1) treat the 3d capturevolume as a scalar field; (2) use a “Marching Cubes” (or a related“Marching Tetrahedrons”) algorithm to find the isosurface of the scalarfield and create a polygon mesh representing the surface of the subject;and (3) remove false surfaces and simplify the mesh. Details associatedwith each of these steps is provided below.

The scalar value of each point in the capture volume (also called avoxel) is the weighted sum of the scalar values from each camera. Thescalar value for a single camera for points near the reconstructedsurface is the best estimate of the distance of that point to thesurface. The distance is positive for points inside the object andnegative for points outside the object. However, points far from thesurface are given a small negative value even if they are inside theobject.

The weight used for each camera has two components. Cameras that lie inthe general direction of the normal to the surface are given a weightof 1. Cameras that lie 90 degrees to the normal are given a weight of 0.A function is used of the form: n_(i)=cos² a_(i), where n_(i) is thenormal weighting function, and a_(i) is the angle between the camera'sdirection and the surface normal. This is illustrated graphically inFIG. 23.

The second weighting component is a function of the distance. Thefarther the volume point is from the surface the less confidence thereis in the accuracy of the distance estimate. This weight decreasessignificantly faster than the distance increases. A function is used ofthe form: w_(i)=1/(d_(i) ²+1), where w_(i) is the weight and d_(i) isthe distance. This is illustrated graphically in FIG. 24. This weight isalso used to differentiate between volume points that are “near to” and“far from” the surface. The value of the scalar field for camera i, is afunction of the form: s_(i)=(d_(i)*w_(i)−k*(1−w_(i)))*n_(i), where d_(i)is the distance from the volume point to the surface, w_(i) is thedistance weighting function, k is the scalar value for points “faraway”, and n_(i) is the normal weighting function. This is illustratedgraphically in FIG. 25. The value of the scalar field is the weightedsum of the scalar fields for all cameras: s=sum(s_(i)*w). See, e.g., AVolumetric Method for Building Complex Models from Range Images BrianCurless and Marc Levoy, Stanford University,http://graphics.stanford.edu/papers/volrange/paper 1 level/paper.html,which is incorporated herein by reference.

It should be noted that other known functions with similarcharacteristics to the functions described above may also be employed.For example, rather than a cosine-squared function as described above, acosine squared function with a threshold may be employed. In fact,virtually any other function which produces a graph shaped similarly tothose illustrated in FIGS. 23-25 may be used (e.g., a graph which fallstowards zero at a high angle).

In one embodiment of the invention, the “Marching Cubes” algorithm andits variant “Marching Tetrahedrons” finds the zero crossings of a scalarfield and generates a surface mesh. See, e.g., Lorensen, W. E. andCline, H. E., Marching Cubes: a high resolution 3D surfacereconstruction algorithm, Computer Graphics, Vol. 21, No. 4, pp 163-169(Proc. of SIGGRAPH), 1987, which is incorporated herein by reference. Avolume is divided up into cubes. The scalar field is known or calculatedas above for each corner of a cube. When some of the corners havepositive values and some have negative values it is known that thesurface passes through the cube. The standard algorithm interpolateswhere the surface crosses each edge. One embodiment of the inventionimproves on this by using an improved binary search to find the crossingto a high degree of accuracy. In so doing, the scalar field iscalculated for additional points. The computational load occurs onlyalong the surface and greatly improves the quality of the resultingmesh. Polygons are added to the surface according to tables. The“Marching Tetrahedrons” variation divides each cube into sixtetrahedrons. The tables for tetrahedrons are much smaller and easier toimplement than the tables for cubes. In addition, Marching Cubes has anambiguous case not present in Marching Tetrahedrons.

The resulting mesh often has a number of undesirable characteristics.Often there is a ghost surface behind this desired surface. There areoften false surfaces forming a halo around the true surface. And finallythe vertices in the mesh are not uniformly spaced. The ghost surface andmost of the false surfaces can be identified and hence removed with twosimilar techniques. Each vertex in the reconstructed surface is checkedagainst the range information from each camera. If the vertex is closeto the range value for a sufficient number of cameras (e.g., 1-4cameras) confidence is high that this vertex is good. Vertices that failthis check are removed. Range information generally doesn't exist forevery point in the field of view of the camera. Either that point isn'ton the surface or that part of the surface isn't painted. If a vertexfalls in this “no data” region for too many cameras (e.g., 1-4 cameras),confidence is low that it should be part of the reconstructed surface.Vertices that fail this second test are also removed. This test makesassumptions about, and hence restrictions on, the general shape of theobject to be reconstructed. It works well in practice for reconstructingfaces, although the underlying principles of the invention are notlimited to any particular type of surface. Finally, the spacing of thevertices is made more uniform by repeatedly merging the closest pair ofvertices connected by an edge in the mesh. The merging process isstopped when the closest pair is separated by more than some thresholdvalue. Currently, 0.5 times the grid spacing is known to provide goodresults.

FIG. 26 is a flowchart which provides an overview of foregoing process.At 2601, the scalar field is created/calculated. At 2602, the marchingtetrahedrons algorithm and/or marching cubes algorithm are used todetermine the zero crossings of the scalar field and generate a surfacemesh. At 2603, “good” vertices are identified based on the relativepositioning of the vertices to the range values for a specified numberof cameras. The good vertices are retained. At 2604, “bad” vertices areremoved based on the relative positioning of the vertices to the rangevalues for the cameras and/or a determination as to whether the verticesfall into the “no data” region of a specified number of cameras (asdescribed above). Finally, at 2605, the mesh is simplified (e.g., thespacing of the vertices is made more uniform as described above) and theprocess ends.

Vertex Tracking Embodiments

“Vertex tracking” as used herein is the process of tracking the motionof selected points in a captured surfaces over time. In general, oneembodiment utilizes two strategies to tracking vertices. TheFrame-to-Frame method tracks the points by comparing images taken a veryshort time apart. The Reference-to-Frame method tracks points bycomparing an image to a reference image that could have been captured ata very different time or possibly it was acquired by some other means.Both methods have strengths and weaknesses. Frame-to-Frame tracking doesnot give perfect results. Small tracking errors tend to accumulate overmany frames. Points drift away from their nominal locations. InReference-to-Frame, the subject in the target frame can be distortedfrom the reference. For example, the mouth in the reference image mightbe closed and in the target image it might be open. In some cases, itmay not be possible to match up the patterns in the two images becauseit has been distorted beyond recognition.

To address the foregoing limitations, in one embodiment of theinvention, a combination of Reference-to-Frame and Frame to Frametechniques are used. A flowchart describing this embodiment isillustrated in FIG. 27. At 2701, Frame-to-Frame tracking is used to findthe points within the first and second frames. At 2703, process variableN is set to 3 (i.e., representing frame 3). Then, at 2704,Reference-to-Frame tracking is used to counter the potential driftbetween the frames. At 2705, the value of N is increased (i.e.,representing the Nth frame) and, if another frame exists, determined at2706, the process returns to 2703 where Frame-to-Frame tracking isemployed followed by Reference-to-Frame tracking at 2704.

In one embodiment, for both Reference-to-Frame and Frame-to-Frametracking, the camera closest to the normal of the surface is chosen.Correlation is used to find the new x,y locations of the points. See,e.g., APPARATUS AND METHOD FOR PERFORMING MOTION CAPTURE USING A RANDOMPATTERN ON CAPTURE SURFACES,” Ser. No. 11/255,854, Filed Oct. 20, 2005,for a description of correlation techniques that may be employed. The zvalue is extracted from the reconstructed surface. The correlationtechnique has a number of parameters that can be adjusted to find asmany points as possible. For example, the Frame-to-Frame method mightsearch for the points over a relatively large area and use a largewindow function for matching points. The Reference-to-Frame method mightsearch a smaller area with a smaller window. However, it is often thecase that there is no discernible peak or that there are multiple peaksfor a particular set of parameters. The point cannot be tracked withsufficient confidence using these parameters. For this reason, in oneembodiment of the invention, multiple correlation passes are performedwith different sets of parameters. In passes after the first, the searcharea can be shrunk by using a least squares estimate of the position ofa point based on the positions of nearby points that were successfullytracked in previous passes. Care must be taken when selecting the nearbypoints. For example, points on the upper lip can be physically close topoints on the lower lip in one frame but in later frames they can beseparated by a substantial distance. Points on the upper lip are notgood predictors of the locations of points on the lower lip. Instead ofthe spatial distance between points the geodesic distance between pointswhen travel is restricted to be along edges of the mesh is a betterbasis for the weighting function of the least squares fitting. In theexample, the path from the upper lip to the lower lip would go aroundthe corners of the mouth—a much longer distance and hence a greatlyreduced influence on the locations of points on the opposite lip.

FIG. 28 provides an overview of the foregoing operations. In 2801, thefirst set of parameters is chosen. In 2802, an attempt is made to trackvertices given a set of parameters. Success is determined using thecriteria described above. In 2802, the locations of the vertices thatwere not successfully tracked are estimated from the positions ofneighboring vertices that were successfully tracked. In 2804 and 2805,the set of parameters is updated or the program is terminated. Thus,multiple correlation passes are performed using different sets ofparameters.

At times the reconstruction of a surface is imperfect. It can have holesor extraneous bumps. The location of every point is checked byestimating its position from its neighbor's positions. If the trackedlocation is too different it is suspected that something has gone wrongwith either the tracking or with the surface reconstruction. In eithercase the point is corrected to a best estimate location.

Retrospective Tracking Marker Selection

Many prior art motion capture systems (e.g. the Vicon MX40 motioncapture system) utilize markers of one form or another that are attachedto the objects whose motion is to be captured. For example, forcapturing facial motion one prior art technique is to glueretroreflective markers to the face. Another prior art technique tocapture facial motion is to paint dots or lines on the face. Since thesemarkers remain in a fixed position relative to the locations where theyare attached to the face, they track the motion of that part of the faceas it moves.

Typically, in a production motion capture environment, locations on theface are chosen by the production team where they believe they will needto track the facial motion when they use the captured motion data in thefuture to drive an animation (e.g. they may place a marker on the eyelidto track the motion of blinking). The problem with this approach is thatit often is not possible to determine the ideal location for the markersuntil after the animation production is in process, which may be monthsor even years after the motion capture session where the markers werecaptured. At such time, if the production team determines that one ormore markers is in a sub-optimal location (e.g. located at a location onthe face where there is a wrinkle that distorts the motion), it is oftenimpractical to set up another motion capture session with the sameperformer and re-capture the data.

In one embodiment of the invention users specify the points on thecapture surfaces that they wish to track after the motion capture datahas been captured (i.e. retrospectively relative to the motion capturesession, rather than prospectively). Typically, the number of pointsspecified by a user to be tracked for production animation will be farfewer points than the number of vertices of the polygons captured ineach frame using the surface capture system of the present embodiment.For example, while over 100,000 vertices may be captured in each framefor a face, typically 1000 tracked vertices or less is sufficient formost production animation applications.

For this example, a user may choose a reference frame, and then select1000 vertices out of the more than 100,000 vertices on the surface to betracked. Then, utilizing the vertex tracking techniques describedpreviously and illustrated in FIGS. 27 and 28, those 1000 vertices aretracked from frame-to-frame. Then, these 1000 tracked points are used byan animation production team for whatever animation they choose to do.If, at some point during this animation production process, theanimation production team determines that they would prefer to have oneor more tracked vertices moved to different locations on the face, or tohave one or more tracked vertices added or deleted, they can specify thechanges, and then using the same vertex tracking techniques, these newvertices will be tracked. In fact, the vertices to be tracked can bechanged as many times as is needed. The ability to retrospectivelychange tracking markers (e.g. vertices) is an enormous improvement overprior approaches where all tracked points must be specifiedprospectively prior to a motion capture session and can not be changedthereafter.

Embodiments of the invention may include various steps as set forthabove. The steps may be embodied in machine-executable instructionswhich cause a general-purpose or special-purpose processor to performcertain steps. Various elements which are not relevant to the underlyingprinciples of the invention such as computer memory, hard drive, inputdevices, have been left out of the figures to avoid obscuring thepertinent aspects of the invention.

Alternatively, in one embodiment, the various functional modulesillustrated herein and the associated steps may be performed by specifichardware components that contain hardwired logic for performing thesteps, such as an application-specific integrated circuit (“ASIC”) or byany combination of programmed computer components and custom hardwarecomponents.

Elements of the present invention may also be provided as amachine-readable medium for storing the machine-executable instructions.The machine-readable medium may include, but is not limited to, flashmemory, optical disks, CD-ROMs, DVD ROMs, RAMs, EPROMs, EEPROMs,magnetic or optical cards, propagation media or other type ofmachine-readable media suitable for storing electronic instructions. Forexample, the present invention may be downloaded as a computer programwhich may be transferred from a remote computer (e.g., a server) to arequesting computer (e.g., a client) by way of data signals embodied ina carrier wave or other propagation medium via a communication link(e.g., a modem or network connection).

Throughout the foregoing description, for the purposes of explanation,numerous specific details were set forth in order to provide a thoroughunderstanding of the present system and method. It will be apparent,however, to one skilled in the art that the system and method may bepracticed without some of these specific details. Accordingly, the scopeand spirit of the present invention should be judged in terms of theclaims which follow.

1. A computer-implemented method implemented within a motion capturesystem for performing surface reconstruction of a subject having athree-dimensional capture volume from multiple-range data setscomprising, the motion capture system comprising a plurality of cameras:creating a scalar field for the three-dimensional (3-D) capture volumeof the subject; generating a surface mesh for the scalar field;retaining good vertices and removing bad vertices of the surface mesh;and storing the good vertices for use in subsequent reconstruction ofthe motion of the subject.
 2. The method as in claim 1 wherein creatinga scalar field comprises: calculating a scalar value for each point inthe capture volume.
 3. The method as in claim 2 wherein each scalarvalue is calculated as the weighted sum of the scalar values calculatedfrom each camera used in a motion capture session.
 4. The method as inclaim 3 wherein the weighted values used for each camera are based on(1) the angle of the camera's direction and the surface normal for thesurface on which each point resides and (2) the distance of the camerafrom each point.
 5. The method as in claim 1 wherein generating asurface mesh comprises: executing a Marching Cubes or MarchingTetrahedrons algorithm to determine zero crossings of the scalar field.6. The method as in claim 1 wherein “good” vertices are identified basedon the relative positioning of the vertices to range values for aspecified number of cameras and “bad” vertices are removed based on therelative positioning of the vertices to the range values for the camerasand/or a determination as to whether the vertices fall into a “no data”region of a specified number of cameras.
 7. The method as in claim 1further comprising: simplifying the surface mesh by repeatedly mergingthe closest pair of vertices connected by an edge in the surface mesh.8. The method as in claim 7 further comprising: discontinuing mergingwhen the closest pair of vertices is separated by more distance than aspecified threshold value.
 9. The method as in claim 8 wherein thethreshold value comprises 0.5 times the grid spacing of the surfacemesh.
 10. A method for performing motion capture of a subjectcomprising: capturing a series of image frames of the subject over aperiod of time each frame each frame having a plurality of verticesdefining a captured surface of the subject; establishing a referenceframe having one or more of the plurality of vertices; performingframe-to-frame tracking to identify vertices within the N′th frame basedon the (N−1)′th frame or an earlier frame; and performingreference-to-frame tracking to identify vertices within the N′th framebased on the reference frame to counter potential drift between theframes.
 11. The method as in claim 10 further comprising: selecting acamera closest to a normal of the surface on which each vertex islocated to perform the frame-to-frame and reference-to-frame tracking.12. The method as in claim 10 wherein the frame-to-frame tracking isperformed using a relatively larger window for matching vertices and thereference-to-frame tracking is performed using a relatively smallerwindow for matching vertices.
 13. The method as in claim 10 furthercomprising performing the frame-to-frame and reference-to-frame trackingagain using a different set of parameters, the parameters defining asearch area for the vertices of each frame.
 14. The method as in claim10 further comprising: estimating the location of vertices not found ineach frame N based on known locations of neighboring vertices.
 15. Acomputer-implemented method for capturing the motion of a subjectcomprising: capturing motion capture data including a plurality ofimages of the N vertices during a motion capture session;retrospectively identifying X of the N vertices to track across theplurality of images where X<N; and tracking the X vertices across theplurality of images.
 16. The method as in claim 15 further comprising:retrospectively identifying Y of the N vertices to track across theplurality of images where Y<N and wherein the Y vertices includevertices not included in the X vertices; and tracking the Y verticesacross the plurality of images.